Created by: gwideman, Jan 31, 2012 9:33 pm Revised by: gwideman, Aug 10, 2012 12:26 pm (19 revisions)
Overview
This article discusses Delphi's TWebBrowser, which is a wrapper for Microsoft's ShDocVW.dll WebBrowser control. In particular, the article pursues the topic of how to tame certain aspects of WebBrowser's behavior. This leads to a revised component, TGwWebBrowserTamed, which implements properties to disable features and behaviors which an application may want to avoid.
Delphi's TWebBrowser might be an attractive component for an application which needs to:
retrieve html pages from a web server
parse each page into a Document Object Model (DOM) tree, which the application can navigate to gather data of interest.
TWebBrowser, in unit ShDocVw, basically wraps the functionality provided by Internet Explorer (IWebBrowser2), notably including the parsing apparatus associated with MSHTML and its DOM.
The diagram to the right, borrowed from MSDN, sketches the architecture.
This would appear to fulfill the requirements I set out above. However, the MS WebBrowser includes a whole lot of other behavior that our custom application may not want:
Downloading additional files that aren't needed, such as images, video, scripts, CSS and so on.
Running scripts (ie: Javascript and VBScript)
Playing video and audio
Throwing errors and/or putting up dialogs
Behaviors which vary, depending on the user's Internet settings.
These raise the question of how to disable or otherwise hem in the behaviors that aren't needed in a particular scenario.
Background
It would be convenient for component users if an application could control the issues listed above by manipulating some option settings exposed by TWebBrowser component, or even by navigating the object model into the underlying COM objects and setting some properties. Unfortunately, it's a bit more complicated than that.
The main issue is that, in the IE architecture, these issues are controlled by settings which the underlying web browser component must request from the containing application, settings referred to as "Ambient Properties" -- that is to say, properties supplied by the "ambient environment" surrounding the control.
The MS WebBrowser control architecture defines the interfaces through which the hosting application and the WebBrowser communicate. The WebBrowser implements a number of interfaces, and the host application or container implements the corresponding host ones. The main topic of of this article is the set of interfaces that the host should implement in order to supply the ambient property values.
Of special interest is the ambient property called "DLCONTROL", or "download control". This is a single integer whose individual bits control whether WebBrowser will automatically download files such as images, videos and scripts, and also whether WebBrowser will execute scripts of various types, and a variety of other behaviors.
Delphi's TWebBrowser, being a wrapper around MS WebBrowser, implements the basic apparatus for hosting the MS WebBrowser. However, it does not implement the ambient properties interface. The revised component described here will fill that gap.
What the DLCONTROL ambient properties control
The following table lists the functions of the individual bits within the DLCONTROL ambient variable expected by WebBrowser. This is essentially the same list provided by MSDN, with additional notes. The listing is in alphabetical order of constant name, not numerical order.
DLCTL_xxx constant
Description
DLCTL_BGSOUNDS
On = play doc's background sounds
DLCTL_DLIMAGES
On = Download images from the server.
DLCTL_DOWNLOADONLY
On = Download the page, but not display it.
DLCTL_FORCEOFFLINE
On = The browsing component will always operate in offline mode. This causes the BINDF_OFFLINEOPERATION flag to be set even if the computer is connected to the Internet when making requests through URLMON.
On = Do not perform any client pull operations. This relates to HTML META tag with attributes HTTP-EQUIV="REFRESH" CONTENT=10 or similar HTML The Definitive Guide
DLCTL_NO_DLACTIVEXCTLS
On = Do not download any ActiveX Controls in the document.
DLCTL_NO_FRAMEDOWNLOAD
On = Do not download frames but do download and parse the frameset page. Ignore the frameset, and render no frame tags.
On = Do not execute any ActiveX Controls in the document.
DLCTL_NO_SCRIPTS
On = Do not execute any scripts.
DLCTL_OFFLINE
Same as DLCTL_OFFLINEIFNOTCONNECTED.
DLCTL_OFFLINEIFNOTCONNECTED
On = Operate in offline mode if not connected to the Internet. This causes the BINDF_GETFROMCACHE_IF_NET_FAIL flag to be set if the computer is connected to the Internet when making requests through URLMON.
DLCTL_PRAGMA_NO_CACHE
On = Do not use Proxy's cache. Instead, force the request through to the server and ignore the proxy, even if the proxy indicates that the data is up to date. This causes the BINDF_PRAGMA_NO_CACHE flag to be set when making requests through URLMON.
DLCTL_RESYNCHRONIZE
On = The browsing component will ignore what is in the cache and ask the server for updated information. The cached information will be used if the server indicates that the cached information is up to date. This causes the BINDF_RESYNCHRONIZEflag to be set when making requests through URLMON. (Here "the cache" presumably refers to local cache, as opposed to a proxy cache involved in DLCTL_PRAGMA_NO_CACHE.)
DLCTL_SILENT
On = Do not display any user interface. This causes the BINDF_SILENTOPERATION flag to be set when making requests through URLMON. (I am hoping this means do not show message boxes and so on.)
DLCTL_URL_ENCODING_DISABLE_UTF8
On = Disable UTF-8 encoding.
DLCTL_URL_ENCODING_ENABLE_UTF8
On = Enable UTF-8 encoding. (What happens if both Enable and Disable are On, or both Off?)
DLCTL_VIDEOS
On = Play any video clips that are contained in the document.
Communication involved in Ambient Properties
Delphi's TWebBrowser implements a container for MS's WebBrowser container. We want to revise TWebBrowser so that it implements the host part of the communication required for WebBrowser to request ambient property values. So our first task is to understand the requirements and steps in that communication.
I found numerous discussions of this topic on the web, at MSDN and elsewhere, but all seemed to be muddled, off-target or incomplete in one way or another, or obfuscated by arcane COM terminology. I derived the following from an example called walkall.cpp (see references), and this turned out to be sufficient to implement the necessary additions to TWebBrowser. Hopefully this will clarify the matters for others too.
As a preliminary, for ambient properties, the container (in our case TWebBrowser or descendant) must implement the following interfaces and methods:
Interface
Method
In TWebBrowser, implemented by
Comment
IOleClientSite
Ancestor TOleControl
Provided by TWebBrowser to WebBrowser as first method of communicating.
QueryInterface
Ancestor TOleControl
Needs to supply TWebBrowser's IDispatch to WebBrowser
IDispatch
Ancestor TOleControl
Invoke
Ancestor TOleControl
We will need to override this method's implementation
In short, TWebBrowser already has implementations of the needed interfaces, but we need to revise TWebBrowser's inherited IDispatch.Invoke so that it can respond to WebBrowser's requests for ambient properties, or at least the ambient property of most interest, DLCONTROL.
Here the communication steps are shown as an activity diagram in tabular format:
Based on the interactions just described, the complete agenda for our revised control is:
Item
Description
Add AmbientDLCtlSet property to TWebBrowser
Add a published property to TWebBrowser which can be set at designtime or runtime. Most conveniently this should be of type set, so that the individual settings can be switched individually in the IDE.
Add getter and setter methods.
Setter should call WebBrowser's OnAmbientPropertyChange method to prompt WebBrowser to fetch new value.
Revise IDispatch.Invoke
Add override for TWebBrowser.Invoke which will add handling for DispID = DISPID_AMBIENT_DLCONTROL, otherwise call inherited Invoke
Diagnostic flag
Because I wasn't entirely confident in whether or when WebBrowser would respond to OnAmbientPropertyChange with a call back for values, I added a flag by which to monitor this.
Component and sample code
The requirements just listed are implemented in the code available for download here:
TGwWebBrowserTamed component which can be installed into the IDE. The project is for Delphi XE2 but should require only changes of project settings to work with previous versions of Delphi.
WebBrowserTest01.exe
Sample program demonstrating creating TGwWebBrowserTamed at runtime. This is useful to try the component without installing, or to make quick modifications.
Includes checkboxes in the UI for exercising each of the DLControl settings live
WebBrowserTest02.exe
Sample program demonstrating creating TGwWebBrowserTamed at designtime. This is useful to try out the designtime settings.
Includes checkboxes in the UI for exercising each of the DLControl settings live
Exe included (see Demos\WebBrowserTest02\Win32\Release)
Instructions
To browse the web: Enter a URL into the edit slot, and then press Navigate.
To change DLCONTROL settings, change some checkboxes and then press "Push to WB"
walkall.exe sample code. Demonstrates how to walk the DOM presented by MSHTML. Comments in the main source file explain how to set ambient properties, (but there is no actual demo code of this.)
mshtmdid.h Header file listing DISPID_xxx and DLCTL_xxx constants
Includes a primer on Control/Container interaction, and discusses IOleControlSite and IOleClientSite.
Discusses ambient properties, including:
"The container lays out the contained control, manages keyboard interaction, enables the properties of controls to be saved in some persistent format, handles events generated by controls, and exposes ambient properties to controls. Ambient properties allow OLE controls to retrieve information about the control site provided by their container."
"[...] Likewise, a container-provided IDispatch enables controls to access container ambient properties and alert the container of events."
"container ambient properties are provided by an IDispatch interface that can be retrieved from the IOleControlSite interface that hosts the control" (Note: "retrieved from" = using QueryInterface, not the IDispatch that can be obtained from IOleControlSite.GetExtendedControl
delphidabbler (Peter Johnson) Several relevant articles with sample code
Source and dcus supplied with XE2, possibly since XE? Eg: $(BDS)\Source\Internet\WebBrowserEx.pas
Not sure what these are, as they aren't components installed in the IDE. Are they working code, or vestigial? Maybe part of a tool application? Appear to tackle operating TWebBrowser in edit mode.
Demonstrates how to override TOleControl's IDispatch.Invoke, to incorporate the DLCTL_xxx ambient properties, and also provide other responses.
"MSHTML" is discussed in narrative as though it is a tangible class or object: "use of MSHTML", "as MSHTML loads the document", "MSHTML's READYSTATE" etc. While it may be a coherent thing behind the scenes, it's actually engaged by:
Jim Beveridge "report from the front" using MSHTML
I belatedly discovered this article on Jim Beveridge's blog: How to load MSHTML data He evidently discovered many of the same articles as I have, but has gone further (and probably smarter) implementing several solutions, and run into various difficulties.
JB tried a variety of different methods to employ larger and smaller chunks of MSHTML to implement html-to-editable-DOM. My summaries of his notes, as reminders to myself:
1. IMarkupServices.ParseString
ParseString wants Unicode. Instead use ParseGlobal for any character set..
Returned IHtmlDocument2 object not fully functional. Lacks serialization.
IDocument3.getDocumentElement fails.
2. IHtmlDocument2.write()
write() requires Unicode.
Needs a SAFEARRAY at some point?
3. IPersistStreamInit.Load() to load HTML (get IPersistStreamInit from QI of any interface on MSHTML, I assume)
Revised by: gwideman, Aug 10, 2012 12:26 pm (19 revisions)
Overview
This article discusses Delphi's TWebBrowser, which is a wrapper for Microsoft's ShDocVW.dll WebBrowser control. In particular, the article pursues the topic of how to tame certain aspects of WebBrowser's behavior. This leads to a revised component, TGwWebBrowserTamed, which implements properties to disable features and behaviors which an application may want to avoid.Delphi's TWebBrowser might be an attractive component for an application which needs to:
- retrieve html pages from a web server
- parse each page into a Document Object Model (DOM) tree, which the application can navigate to gather data of interest.
TWebBrowser, in unit ShDocVw, basically wraps the functionality provided by Internet Explorer (IWebBrowser2), notably including the parsing apparatus associated with MSHTML and its DOM.The diagram to the right, borrowed from MSDN, sketches the architecture.
This would appear to fulfill the requirements I set out above. However, the MS WebBrowser includes a whole lot of other behavior that our custom application may not want:
- Downloading additional files that aren't needed, such as images, video, scripts, CSS and so on.
- Running scripts (ie: Javascript and VBScript)
- Playing video and audio
- Throwing errors and/or putting up dialogs
- Behaviors which vary, depending on the user's Internet settings.
These raise the question of how to disable or otherwise hem in the behaviors that aren't needed in a particular scenario.Background
It would be convenient for component users if an application could control the issues listed above by manipulating some option settings exposed by TWebBrowser component, or even by navigating the object model into the underlying COM objects and setting some properties. Unfortunately, it's a bit more complicated than that.The main issue is that, in the IE architecture, these issues are controlled by settings which the underlying web browser component must request from the containing application, settings referred to as "Ambient Properties" -- that is to say, properties supplied by the "ambient environment" surrounding the control.
The MS WebBrowser control architecture defines the interfaces through which the hosting application and the WebBrowser communicate. The WebBrowser implements a number of interfaces, and the host application or container implements the corresponding host ones. The main topic of of this article is the set of interfaces that the host should implement in order to supply the ambient property values.
Of special interest is the ambient property called "DLCONTROL", or "download control". This is a single integer whose individual bits control whether WebBrowser will automatically download files such as images, videos and scripts, and also whether WebBrowser will execute scripts of various types, and a variety of other behaviors.
Delphi's TWebBrowser, being a wrapper around MS WebBrowser, implements the basic apparatus for hosting the MS WebBrowser. However, it does not implement the ambient properties interface. The revised component described here will fill that gap.
What the DLCONTROL ambient properties control
The following table lists the functions of the individual bits within the DLCONTROL ambient variable expected by WebBrowser. This is essentially the same list provided by MSDN, with additional notes. The listing is in alphabetical order of constant name, not numerical order.Communication involved in Ambient Properties
Delphi's TWebBrowser implements a container for MS's WebBrowser container. We want to revise TWebBrowser so that it implements the host part of the communication required for WebBrowser to request ambient property values. So our first task is to understand the requirements and steps in that communication.I found numerous discussions of this topic on the web, at MSDN and elsewhere, but all seemed to be muddled, off-target or incomplete in one way or another, or obfuscated by arcane COM terminology. I derived the following from an example called walkall.cpp (see references), and this turned out to be sufficient to implement the necessary additions to TWebBrowser. Hopefully this will clarify the matters for others too.
As a preliminary, for ambient properties, the container (in our case TWebBrowser or descendant) must implement the following interfaces and methods:
Here the communication steps are shown as an activity diagram in tabular format:
<--
-->
Agenda for TGwWebBrowserTamed
Based on the interactions just described, the complete agenda for our revised control is:Component and sample code
The requirements just listed are implemented in the code available for download here:Features included
Instructions
References
Appendices
Using MSHTML directly instead of WebBrowser
Using IMarkupServices directly instead of WebBrowser
Jim Beveridge "report from the front" using MSHTML
- I belatedly discovered this article on Jim Beveridge's blog: How to load MSHTML data He evidently discovered many of the same articles as I have, but has gone further (and probably smarter) implementing several solutions, and run into various difficulties.
- JB tried a variety of different methods to employ larger and smaller chunks of MSHTML to implement html-to-editable-DOM. My summaries of his notes, as reminders to myself:
- 1. IMarkupServices.ParseString
- ParseString wants Unicode. Instead use ParseGlobal for any character set..
- Returned IHtmlDocument2 object not fully functional. Lacks serialization.
- IDocument3.getDocumentElement fails.
- 2. IHtmlDocument2.write()
- write() requires Unicode.
- Needs a SAFEARRAY at some point?
- 3. IPersistStreamInit.Load() to load HTML (get IPersistStreamInit from QI of any interface on MSHTML, I assume)
- Loading HTML content from a Stream
- Requires CoInitializeEx(NULL,COINIT_MULTITHREADED)
- Recent versions of MSHTML require a message loop to get the work done.
- Bug in MIME interpretation, but fixed in IE7 and later
- 4. IPersistFile.Load() (interface to HTML doc)
- Could load html file from memory, if it was in a file. But also requires message loop.
- 5. IPersistMoniker to feed the stream to MSHTML
- No need for Unicode buffer, can use in-memory data
My digestion: Aside from trying to troubleshoot and work around actual bugs, JB's main concerns seem to have been: