Tuesday, May 29, 2007

Has a File Been Virtualized?

If you've been working with or reading about Windows Vista, you know there's a new feature called virtualization that causes writes to files in "protected" folders (such as C:\Program Files\Some Folder) to be redirected to a user-specific folder (C:\Users\user name\AppData\Local\VirtualStore\Program Files\Some Folder). While this is sort of a good thing--it means pre-Vista applications won't give an error when they update data stored in their own directory--it can be a problem for a variety of reasons:
  • Since the files are redirected to a user-specific location, other users on the same system won't see each others' changes.
  • Backing up the files in the application folder won't back up the user-specific files if the backup program is Vista-aware.
  • If the user turns off User Access Control (UAC) or virtualization, the application not only breaks because it still can't write to the protected folders, it also doesn't see the virtualized files any more so data entered by the user appears to be gone.

So, I've been reworking those parts of Stonefield Query that expected to write to the current or application folder. It was actually fairly easy to do because most file paths were referenced with an application object property (e.g. USE (oApp.cAppDataDir) + "SomeTable") so simply changing the value of the property redirected the files to a location the user has write access to. I then ran the application and looked for any files that appeared in my virtualization folder, and turned off virtualization and tested to see if anything broke. I found a few stragglers where I hadn't put a path on the filename (typically writing to some temporary files) and fixed them.

However, one thing puzzled me: a table that wasn't being written to was virtualized. In tracking it down, I found that the virtualized table was created immediately after the USE command. To make matters even odder, when I turn off virtualization, the table wasn't virtualized and the application didn't give an error. Also, while the DBF and FPT files were virtualized, the CDX was not.

I guess this makes sense. Since I was opening the table in read-write mode (ie. no NOUPDATE clause), I suspect VFP was doing some minor update to the table behind the scenes (e.g. writing to the header of DBF and FPT files for locking purposes), resulting in it being virtualized. You can see this effect if you open one of the classes in the FFC subdirectory of the VFP home directory; simply opening it immediately causes its virtualization. With virtualization turned off, the table was automatically read-only since it's in a protected folder, so there's no need to virtualize it. Adding a NOUPDATE clause to the table (since it's a meta data table, it's used for reference purposes only) took care of that.

However, the SDK for Stonefield Query includes an editor program for that table, so that program does open the table for read-write purposes and will likely write to the table (otherwise it wouldn't be an editor). The problem is that the user may be unaware that the table is virtualized and will likely distribute the wrong instance of the table (the one in the application folder rather than the virtualized one) after making changes to it. So, I figured I should let them know which table they're writing to so they know which one to distribute.

However, how can you tell if a file has been virtualized? If you open a file in C:\Program Files\Some Folder and that file has been virtualized, the operating system actually opens the virtualized copy instead of the one you specified, and the application can't tell the difference.

I settled on the obvious solution: check whether a virtualized copy of the file in question exists. If so, we know that's the one Vista will open. Here's some code to do that. First, we get the directory the application is running in (not necessarily the current directory) by calling GetAppDirectory (a routine I wrote years ago; I'll show you the code later). Then we call SpecialFolders, a routine I discussed in my Finding the Paths for Special Folders blog entry, to get the current user's local app data path (C:\Users\user name\AppData\Local\). Notice that this code uses SUBSTR(lcDirectory, 3) to strip the drive letter, colon, and backslash from the application folder name. We then check if the desired file exists in the virtualization folder.

lcDirectory   = GetAppDirectory()
lcVirtualPath = SpecialFolders('LocalAppData') + ;
'VirtualStore\' + substr(lcDirectory, 3)
lcFile = lcVirtualPath + 'SomeTable.DBF'
if file(lcFile)
* the table has been virtualized

Here's the code for GetAppDirectory. It handles runtime and development time conditions differently.

local lcPath
if version(2) = 2
lcPath = justpath(sys(16, 0))
if atc('PROCEDURE', lcPath) > 0
lcPath = substr(lcPath, rat(':', lcPath) - 1)
endif atc('PROCEDURE', lcPath) > 0
lcPath = addbs(justpath(_vfp.ServerName))
endif version(2) = 2
return lcPath

Thursday, May 03, 2007

Why I Signed

I signed the petition today.

All the negative comments from some people in the community had done more to convince me not to sign than anything else. I'm a pretty laid back guy, but I get really pissed off when people question my integrity.

Few of the comments on my "Why I Haven't Signed" blog entry or the Wiki topic actually tried to make an argument. Most of them amounted to "you're wrong" (that's your idea of a convincing argument?) and "VFP is a great tool" (thanks, know that already).

However, Mike Asherman's comment made me think about this some more. He said, "Adding your name means that you support the cause, not that you believe it is likely to succeed." Good point. I've voted for people I've believed in before knowing they didn't have a chance in winning because it was the right thing to do.

So, I signed as my way of showing solidarity with the community. I still believe this petition is futile and will have absolutely no impact on Microsoft's decision, but I've cast my ballot anyway.

For those of you questioning the reasons why others haven't signed, I suggest spending your time in more productive manners, such as writing code and helping others on community forums, rather than driving the wedge into our community even further. Alienating people by questioning their motives is a sure way to get the opposite of what you desire.

Wednesday, May 02, 2007

Taking out the Slow Parts

Sage Timberline Office, like many accounting systems, allows you to customize the captions that appear on-screen for most fields. To make it easier for the user, it makes sense for Stonefield Query for Sage Timberline Office, the newest member in the Stonefield Query family of products, to display the user's own captions. However, one slight complication is that they aren't stored in a manner that's easily readable. Instead, we have to call a COM object to retrieve the captions. This object returns XML specifying the name of each field and its caption. Given that there are tens of thousands of fields in Timberline, this XML can be over 1 MB in size. Here's what the XML looks like:

<fieldcaption>Caption for Field</fieldcaption>
We want to use this XML to update the Stonefield Query data dictionary. Our first attempt used XMLTOCURSOR() to put the XML into a cursor, which we then process. It took somewhere around 30 seconds, which seems like forever because it happens every time the user starts Stonefield Query. XMLTOCURSOR() uses the XML DOM object, which gets exponentially slower as the XML gets larger.

The next approach was to manually parse the XML using STREXTRACT(). STREXTRACT() is the perfect function for parsing XML because it was designed to find text between delimiters. If you know the XML structure, it's really easy to use. Here's the code we used:

create cursor __CAPTIONS (FIELDNAME C(119), CAPTION C(60))
for lnI = 1 to occurs('<tablename>', lcXML)
lcTable = strextract(lcXML, '<tablename>', '</tablename>', lnI)
lcField = strextract(lcXML, '<fieldname>', '</fieldname>', lnI)
lcCaption = strextract(lcXML, '<fieldcaption>', '</fieldcaption>', lnI)
insert into __CAPTIONS values (lcTable + '.' + lcField, lcCaption)
next lnI
Interestingly, this turned out to be way slower than XMLTOCURSOR() (in fact, I stopped it after a couple of minutes, so I don't know how long it would've taken).

It occurred to me while looking at the XML that if we could convert it into comma-delimited text, we could use APPEND FROM to suck it into the cursor. Here's the code for that:

#define ccCRLF chr(13) + chr(10)
lcXML = strtran(lcXML, '<tablename>')
lcXML = strtran(lcXML, '</tablename>' + ccCRLF + '<fieldname>', '.')
lcXML = strtran(lcXML, '</fieldname>' + ccCRLF + '<fieldcaption>', ',"')
lcXML = strtran(lcXML, '</fieldcaption>' + ccCRLF + '</newrecord>' + ;
ccCRLF + '<newrecord>', '"')
lcXML = strtran(lcXML, '</fieldcaption>' + ccCRLF + '</newrecord>' + ;
ccCRLF + '</vfpdata>', '"')
lcTemp = forcepath(sys(2015) + '.txt', sys(2023))
strtofile(substr(lcXML, 91), lcTemp)
create cursor __CAPTIONS (FIELDNAME C(119), CAPTION C(60))
append from (lcTemp) delimited
erase (lcTemp)
This took 0.7 seconds, 42 times faster than using XMLTOCURSOR().

But still, I wondered why the STREXTRACT() approach was so slow; my experience is that the VFP text handling functions are blindingly fast. I wondered if it had to do the with fourth parameter, the one that specified the occurrence. I rewrote the code to this:
create cursor __CAPTIONS (FIELDNAME C(119), CAPTION C(60))
lnLines = alines(laLines, lcXML)
for lnI = 1 to lnLines
lcLine = laLines[lnI]
do case
case '<TableName>' $ lcLine
lcTable = strextract(lcLine, '<TableName>', '</TableName>')
case '<FieldName>' $ lcLine
lcField = strextract(lcLine, '<FieldName>', '</FieldName>')
case '<FieldCaption>' $ lcLine
lcCaption = strextract(lcLine, '<FieldCaption>', '</FieldCaption>')
insert into __CAPTIONS values (lcTable + '.' + lcField, lcCaption)
next lnI
This took 0.4 seconds. Wow! Nearly two orders of magnitude improvement over the initial attempt. It really does pay to take out the slow parts!

Off to DevCon

Saturday morning, I'm heading for DevCon in Anaheim, California. Coincidentally, Rick Schummer and I are on the same flight from Minneapolis to Orange County, which may be bad news for him, given my luck with flights leaving on time.

This will be my 17th DevCon as an attendee (I only missed the first one in Toledo in 1989) and my 11th as a speaker (1997 was the first). I'm the only person who will have attended 17 consecutive DevCons, although this will also be number 17 for Alan Griver (just not consecutive; he was at the first one but missed one in the middle).

However, my string of consecutive DevCons as an exhibitor (1993 was the first) is broken: For the first time in fifteen years, I won't have a booth there. Unfortunately, it just wasn't cost effective anymore, plus our booth (and staff) are going to be at the 2007 Timberline User Group National Conference in Dallas at roughly the same time, showing our new Stonefield Query for Timberline.

For posterity, here are the ones I've attended:

1990 Toledo
1991 Toledo
1992 Phoenix
1993 Orlando
1995 San Diego
1996 Scottsdale
1997 San Diego
1998 Orlando
1999 Palm Springs
2000 Miami
2001 San Diego
2002 Ft. Lauderdale
2003 Palm Springs
2004 Las Vegas
2005 Las Vegas
2006 Phoenix
2007 Anaheim

I'm presenting four sessions at DevCon:

  • Best Practices for Vertical Application Development: this is an updated version of the session I presented at Great Lakes Great Database Workshop in 2006 and earlier this year at OzFox. I'll also be presenting this session at Southwest Fox 2007.

  • Integrating RSS and VFP: this is an updated version of the session I gave last year at DevCon. It now includes material on accessing the common RSS store installed with Vista and IE 7.

  • Deploying VFP Applications with InnoSetup: this is an updated version of the session I gave at last year's Southwest Fox and earlier this year at OzFox. It has a new section on deploying on Vista.

  • Developing VFP Applications for Windows Vista: this is a new session and the one I'm most excited about. Vista presents some cool opportunities but also some challenges to all developers, regardless of the language they use. I think it'll be an eye-opener for those who haven't deployed an application on Vista yet or are struggling with the issues right now. I'm also presenting this session at Southwest Fox 2007.

Tuesday, May 01, 2007

Southwest Fox 2007 Speakers and Sessions Announced

The crew at Geek Gatherings are pleased to announce that speakers and sessions have been selected for Southwest Fox 2007. I am really excited about both the lineup of speakers and the sessions they're going to be presenting. This is shaping up to be the must-attend conference of the year.

OK, I'm biased for multiple reasons--both as a speaker and being part of Geek Gatherings--but I really do think this is going to be a killer conference. Some of the sessions, like Alan Stevens' Integrating Windows Forms UI Elements with Existing Applications, Kevin Goff's Professional Business Reporting with Crystal Reports, Christof Wollenhaupt's On the Dark Side of FoxPro, and Toni Feltman's Introduction to the DBI Controls Included in Sedna, are very high on my personal list.

Registration is now open. Be sure to take advantage of the early-bird discount: register by July 1 to save $75 and get a free pre-conference session (a $99 value).