Alternative to NavigationMixin not working in LWC Component that is embedded inside Visualforce page
As we all know post Summer 19 you can add Lightning web components to your Visualforce pages to leverage any investment you’ve made in Visualforce. Implement new functionality using Lightning web components and then use it with existing Visualforce pages. When you look at lightning-navigation, you’ll see it’s only supported in Lightning Experience, Lightning Communities, Salesforce Mobile App. Notably, it is not supported in Lightning Out. You’ll need to come up with a different approach.
So how do you achieve a requirement where you have to navigate back to a record page . It's an easy task using window.location.origin.In the below solution I have created a Visual force tab that contains an LWC component rendered as a modal.

Below is the code snippet to achieve the same.
Step 1: Add the Lightning Components for Visualforce JavaScript Library:Add <apex:includeLightning/>
at the beginning of your Visualforce page. This component loads the JavaScript file used by Lightning Components for Visualforce created.
<apex:page><apex:includeLightning /><div id=”lightningvf” /><script>//Reference a Standalone Aura Dependency App-LtngApp$Lightning.use(“c:LtngApp”, function () {$Lightning.createComponent(“c:calledLWCComponent”,{//API propertyrecordId: “0032v00002nmbzhAAA”
//"{!$CurrentPage.parameters.urlParamValue}"},“lightningvf”,function (cmp) {// do some stuff});});</script></apex:page>
Step 2: Create and Reference a Standalone Aura Dependency App
LtngApp.app
<aura:application extends=”ltng:outApp” access=”GLOBAL”>
<! —
ltng:outApp adds SLDS resources to the page to allow our Lightning components to be
styled with the Salesforce Lightning Design System. If we don’t want SLDS resources added to the page,
we need to extend from ltng:outAppUnstyled instead._ →
<aura:dependency resource=”calledLWCComponent” />
</aura:application>
Step 3: Reference the LWC component in the above-created vf page.
calledLWCComponent
<template><sectionrole=”dialog”tabindex=”-1"aria-labelledby=”modal-heading-01"aria-modal=”true”aria-describedby=”modal-content-id-1"class=”slds-modal slds-fade-in-open”><div class=”slds-modal__container”><! — Modal/Popup Box LWC header here →<header class=”slds-modal__header”><h2id=”modal-heading-01"class=”slds-text-heading_medium slds-hyphenate headerStyle”>LWC Modal</h2></header><! — Modal/Popup Box LWC body starts here →<divclass=”slds-modal__content slds-p-around_medium”id=”modal-content-id-1"><p style=”text-align: center”><strong>Please press continue to navigate to record page. </strong></p></div><! — Modal/Popup Box LWC footer starts here →<footer class=”slds-modal__footer”><! — Add SLDS classes for anchor tag a button experience →<aclass=”slds-button slds-button_brand”href={setBaseUrl}if:true={setBaseUrl}>Continue</a></footer></div></section><div class=”slds-backdrop slds-backdrop_open”></div></template>import { LightningElement, api } from “lwc”;export default class CalledLWCComponent extends LightningElement {//Place holder for URL of SalesforcesfdcBaseURL;//data passed from VF@api recordId = “”;connectedCallback() {console.log(“value for vf-” + this.contact + window.location.origin);}/*for custom URL for use in href of anchor tab*/get setBaseUrl() {return (this.sfdcBaseURL = window.location.origin + “/” + this.recordId);}}
Thats it.You are all set. Happy coding and sharing.
REFERENCES:
https://developer.salesforce.com/docs/component-library/documentation/en/lwc/use_visualforce