I’ll be doing my first ever online presentation hosted by the mighty EMC marketing machine at 12:00 noon EDT on Wednesday, 01/28. I have no idea what to expect from this, in terms of attendance.  However, since I’m leading it, I can tell you what to expect if you make time for it.

I’m going to be describing what we here at EMC mean when we talk about "Enterprise Search."  I’m distinguishing between regular old SharePoint Search (which is quite useful by itself) and Enterprise Search, which is a much bigger, nastier, interesting and perplexing problem to solve.

I’m going to make the argument that FAST is a good tool to use while trying to solve the Enterprise Search problem (and I make some analogies to canals while I’m at it).  This obviously ties into SharePoint and will be a part of the community’s fabric over the course of 2009 and beyond.  That’s the hook for anyone reading my blog here.

If you’re interested in FAST, I think you’ll find enough useful information to justify the time spent listening to me talk about it.

It’s a marketing presentation, but all of the marketing stuff is at the end, so you can always drop off at that point.

The mighty EMC marketing machine has provided me with a signup link, so please use it to register.

</end>

Subscribe to my blog.

Follow me on Twitter at http://www.twitter.com/pagalvin

I had the good fortune of attending some partner training from FAST last spring.  I was really impressed with the product and was looking forward to working with it.  Unfortunately, one project fell through and then EMC acquired my company.  Predictably, a certain amount of chaos ensued while we learned about EMC and EMC learned about us.  FAST technology dropped a few points on the priority scale during that period.  However, I never lost my interest in the product and, more interestingly, the bigger problem of findability. 

I really don’t like that word, but I’m trying to get used to it 🙂  Despite it’s awkwardness, findability is a real (or at least, emerging) term.  Do a live search if you’re interested in finding more technical definitions, but the way I explain it around the office is like this: 

Intellectual capital that cannot be found may as well not exist.

It’s almost as true to say this:

Intellectual capital that cannot be found quickly and easily may as well not exist.

Intellectual capital (IC) starts as an idea in a person’s head and is then refined via collaboration with colleagues and interactions with various communities.  To be truly useful, these resulting ideas must be recorded.  This is where the trouble begins 🙂

These days, recording normally means that the idea is documented in the form of an MS Word doc, Excel workbook, etc. and eventually stuck in electronic format on a hard drive somewhere.  IC obviously takes other forms like, like images, videos, highly informative blogs, wikis … it’s impossible to list them all.  At the same time, IC is stored in a variety of places like file systems, databases, line of business applications (ERP, CRM, SharePoint, Documentum), etc. 

This is the findability problem: how can quickly and easily find IC that is stored in dozens or hundreds of formats in dozens or hundreds of thousands, tens of thousands (dare I say hundreds of thousands) of locations in an organization? 

It’s a difficult problem to solve.  Bill English has been writing about findability from a very grand perspective in what I have come to think of as the Panama Canal approach.  The history of the Panama Canal is amazing.  In a nut shell, a crazy Frenchman (Ferdinand de Lesseps) started a private company to build the canal, the project was abandoned for some years, picked up again and finally finished by the American government under President Roosevelt.  This reminds me of Bill’s approach because as he rightly points out, solving the findability is both hard and never stops.  It took years and years of effort from the some of the hardest working humans on the planet to start, continue, and finally finish).  And yet, it’s still not truly finished.  As far as I know, the canal’s banks have never met their angle of repose, meaning that they have to be shored up and otherwise maintained even to this day.  Solving findability is the same way.  I definitely recommend that you read Bill’s series and subscribe to his blog for his point of view on findability, particularly as it relates to SharePoint.

I too am interested in this problem.  Due to my exposure to FAST and on-going discussions on this subject with my brilliant EMC colleagues, I have some more ideas I plan to write about over the coming weeks and months.  In my next article on this little series, I’m going to try and put a box around the problem to show how awful it really is (it’s more awful than you think 🙂 ).  It’s awful, but at least it does fit inside a box. 

</end>

Subscribe to my blog.

Follow me on Twitter at http://www.twitter.com/pagalvin

I’m starting day 4 of FAST’s partner training headed up by Larry Kaye here in Needham, MA.

This 5-day session is broken down into classes (3 and 2 days respectively) entitled "FAST ESP: Developing Custom Search Applications for Alliance Partners I" and "FAST ESP: Developing Custom Search Applications for Alliance Partners II".

This is a real boot camp type class. The material is deep (very, very deep). The instructor (Larry) clearly knows his stuff. I highly recommend this training if you considering it. 

</end>

This is a continuation in my FAST coding mini-series for SharePoint folk.  The bit of code below executes a search for the term "test" in FAST via a Console application.  Obviously, it’s a hop, skip and jump away to wrap this logic inside a web part or application page:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Collections.Specialized;
using Com.FastSearch.Esp.Search;
using Com.FastSearch.Esp.Search.Http;
using Com.FastSearch.Esp.Search.Navigation;
using Com.FastSearch.Esp.Search.Query;
using Com.FastSearch.Esp.Search.Result;
using Com.FastSearch.Esp.Search.View;
using System.Collections;

namespace Conchango
{
    class InvokeSimpleFASTQuery
    {
        static void Main(string[] args)
        {

            #region Initialize our communication with FAST
            ISearchFactory searchFactory;

            NameValueCollection nameValueCollection = new NameValueCollection();

            nameValueCollection.Add("fastsearchengine", "Com.FastSearch.Esp.Search.Http.HttpSearchFactory");
            nameValueCollection.Add("Com.FastSearch.Esp.Search.Http.QRServers", "fastdemoback:15100");
            nameValueCollection.Add("Com.FastSearch.Esp.Search.Http.RequestMethod", "GET");

            searchFactory = SearchFactory.NewInstance(nameValueCollection);

            #endregion

            #region Launch a query

            string queryString = "test";

            Query query = new Query(queryString);

            query.SetParameter(BaseParameter.CLUSTERING, false);
            query.SetParameter(BaseParameter.NAVIGATION, true);
            query.SetParameter(BaseParameter.NAVIGATION_DEEPHITS, 100);

            ISearchView searchView;
            searchView = searchFactory.GetSearchView("sharepointconn");

            IQueryResult result = searchView.Search(query);

            Console.WriteLine("Total results from search: [" + result.DocCount + "].");

            int docCounter = 0;

            while (docCounter < result.DocCount)
            {

                docCounter++;

                IDocumentSummary thisDocSummary = result.GetDocument(docCounter);

                Console.WriteLine("[" + docCounter + "]: " + thisDocSummary.GetSummaryField("url").StringValue);

            } // while

            Console.ReadLine();
            #endregion

        }
    }
}

It generates the following result:

</end>

Subscribe to my blog.

Here is some code that shows all available views from a FAST server based on the named QRServer (the server process with which we communicate).  I offer this up to give you all a taste of what it’s like to use the provided FAST API.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Com.FastSearch.Esp.Search;
using Com.FastSearch.Esp.Search.Http;
using Com.FastSearch.Esp.Search.Navigation;
using Com.FastSearch.Esp.Search.Query;
using Com.FastSearch.Esp.Search.Result;
using Com.FastSearch.Esp.Search.View;
using System.Collections.Specialized;
using System.Collections;


namespace Conchango
{
    class EnumerateFASTViews
    {
        static void Main(string[] args)
        {

            ISearchFactory searchFactory;

            NameValueCollection nameValueCollection = new NameValueCollection();

            nameValueCollection.Add("fastsearchengine", "Com.FastSearch.Esp.Search.Http.HttpSearchFactory");
            nameValueCollection.Add("Com.FastSearch.Esp.Search.Http.QRServers", "fastdemoback:15100");
            nameValueCollection.Add("Com.FastSearch.Esp.Search.Http.RequestMethod", "GET");

            searchFactory = SearchFactory.NewInstance(nameValueCollection);

            Console.WriteLine("Total Views: [" + searchFactory.GetSearchViewList().Count + "].");

            int i = 0;

            foreach (object o in searchFactory.GetSearchViewList())
            {
                Console.WriteLine("View [" + i++ + "]: [" + o.ToString() + "].");

            }

            Console.ReadLine();

        }
    }
}

Note that I consider myself a barely competent C# programmer and I have thin skin, so don’t send any raspberries my way, especially re: the "foreach (object o in …)" construct 🙂

</end>

Subscribe to my blog.

I’ve been lucky to have an opportunity to work on a FAST <–> SharePoint integration project and thought I’d continue to share my limited experience with the community.

Microsoft is in the process of purchasing FAST.  I don’t know anything about the schedule except that it should be finished in the "2nd quarter" which I assume means by 06/30/08.

My project’s design, at a very high level, calls for creating a few web parts that have roughly the same look, feel and functionality as the standard MOSS search box and core results web part, but pulling from FAST.  I’ve made some baby steps in that direction.

FAST is a dedicated search engine.  It’s not based on Microsoft technology, though it’s obvious to me that some of the FAST engineers clearly understand MS tech (or they’ve made use of contractors). 

FAST:

FAST is a big monster server product (and I don’t mean that in a bad sense; MOSS is a big monster product to me as well).  It appears to be based on Java and I noticed some apache stuff and maybe some PHP.  I wouldn’t be surprised if there were other bits of tech mixed in here and there.  It definitely runs on windows server and is working fine in the virtual environment which I’ve been using.

It makes use of a facade web service that crawls SharePoint content for indexing in its special FAST way. 

A SharePoint end user would be pretty much at home with FAST’s user interface.  It has simple/advance search, shows results, is big on relevancy (though I don’t see how it handles social relevancy).  However, it does go further.  Its stemming seems better.  I really can’t articulate it well but I’ll just say "it’s better."  You can take my word for it, or not 🙂  I may elaborate on this point in the future, as I learn more.

MOSS:

FAST provides a .NET-friendly DLL that provides an interface to the FAST engine.  This makes it possible for us to create web parts or application pages or whatever we want to query FAST and present the results.

There seems to be a web service interface as well.

I also have access to a FAST-provided web part that does integrate MOSS and FAST.  I don’t know if this is a POC, something that was whipped up as a demo or what.  It’s definitely a good learning tool, not so sure if it’s production-ready.

That’s it for now.  As I progress through the project, I’ll post more.

</end>

Subscribe to my blog.

In order to communicate with FAST Search via its API, we need to know the port on which the QRServer is running.

I followed these steps to locate it:

1. Log into the FAST admin site (http://server:port/admin).
2. Navigate to System Overview.
3. Locate "Search Dispatcher" and click the spyglass icon:

   

    

4. I successfully used the value, 15100 for "Info Name" = "http":

   

Notes:

1. Port 15100 appears to be the default out-of-the-box port number for the QRServer. 

2. Use this code similar to the following to instantiate a connection to FAST:

 ISearchFactory searchFactory;

    NameValueCollection nameValueCollection = new NameValueCollection();

    nameValueCollection.Add("fastsearchengine", "Com.FastSearch.Esp.Search.Http.HttpSearchFactory");
    nameValueCollection.Add("Com.FastSearch.Esp.Search.Http.QRServers", "fastdemoback:15102");
    nameValueCollection.Add("Com.FastSearch.Esp.Search.Http.RequestMethod", "GET");

    searchFactory = SearchFactory.NewInstance(nameValueCollection);

3. If you pick a wildly incorrect port, you get a helpful error message:

4. However, if you pick the port number listed on the overview (15102) you don’t get that helpful error message.  Instead, later when you try to access any data via a search or reference a FAST view, you get an error.  Debugging shows that the searchFactory’s view count is zero.  You’ll be given the message "Failed to fetch Search View List".

</end>

Subscribe to my blog.