A Brief History of SharePoint (From a Relative Newcomer’s Perspective)

Note: This article was originally posted to www.endusersharepoint.com.  I forgot to post it to my own blog 🙂

SharePoint has evolved a great deal since its early days as sort of an incubation technology at Microsoft –it’s evolved almost like a horror movie, where the mad scientist’s creation takes on a life of its own, breaking free of its creator’s expectations and rules.  The technical evolution is obvious – the WSS 3.0 object model is richer and more complex than WSS 2.0, which was itself an improvement over earlier versions.  The next version will no doubt show tremendous improvement over 3.0.  From an End User’s perspective, however, SharePoint’s evolution is even more significant.

In the early days, SharePoint didn’t offer much to End Users.  They would have their usual functionality requirements, work with IT to define them well and implement a solution.  IT would use SharePoint to solve the problem.  The product wasn’t very accessible to End Users.  I’ve thought threw a few analogies, but I decided to stick Venn Diagrams to show what I mean.  When Microsoft first released SharePoint to the world as a commercial offering, it followed a relatively traditional pattern of End User <-> IT relationship.  A lot of End Users, communicating and working with a very small number of It people to deliver solutions that solve business problems:

The overall problem domain for which SharePoint is a suitable delivery platform is small (especially compared to today’s SharePoint.  End Users and IT worked in a more classic arrangement with IT: define requirements to IT, wait for IT do their work behind the curtain and take delivery of the final product.

As SharePoint evolved to the 2.0 world (WSS 2.0 and SharePoint Portal Server), several things happened.  First, the “problem domain” increased in size.  By problem domain, I mean the kinds of business problems for which SharePoint could be a viable solution.  For instance, you wouldn’t think too hard about implementing a serious search solution in a SharePoint environment until SPS (and even then, it wasn’t as good as it needed to be).  At the same time, End Users have an unprecedented ability to not only define, but also implement their own solutions with little or no IT support. 

The 3.0 platform (WSS and MOSS) maintained and increased that momentum.  The problem domain is enormous as compared to the 2.0 platform.  Virtually every department in a company, ranging from manufacturing health and safety departments to marketing, from sales to quality control – they can find a good use for SharePoint (and it’s not a case of mashing a round peg into a square hole).  At the same time, the platform empowers even more End Users to implement their own business solutions.  I try to capture that with this diagram:

This has proven to be both a potent and frustrating mixture.  The 3.0 platform turns previously stable roles on their heads.  Suddenly, End Users are effectively judge, jury and executioner business analyst, application architect and developer for their own business solutions.  This gets to the heart of the problem I’m writing about.  But before I dive into that, let’s consider the elephant in the room.

Peering into the Crystal Ball

How will SharePoint 2010 affect this pattern?  Will it be incremental or revolutionary?  Will more, fewer or about the same number of End users find themselves empowered to build solutions in SharePoint 2010?  Will SharePoint 2010’s problem domain expand even further or will it just refine and streamline what it already offers in WSS 3.0 / MOSS?

There’s enough information “out there” to safely say that the general answer is:

• The problem domain is going to dramatically expand. 
• End Users will find themselves even more empowered than before.

The Venn Diagram would be larger than this page and cause some IT Pros and CxO’s to reach for their Pepto. 

I believe it’s going to be a tremendous opportunity for companies to do some truly transformational things. 

No Bulls in My China Shop!

This sounds great, but from my point of view as a SharePoint consultant and putting myself into the shoes of an IT manager, I see this vision.  I own a China shop with beautiful plates, crystal, etc (my SharePoint environment).  I’ve rented a space, I’ve purchased my inventory and laid it all out the way I like it.  I’m not quite ready to open, but in anticipation, I look at the door to see if my customers are lining up and I notice an actual bull out there.  I look more closely and I actually see two bulls and even a wolf.  Then I notice that there are some sheep.  Sheep are so bad, but are they maybe disguised wolves?  I don’t want bulls in my china shop!

It gets worse!  When I rented the space, I couldn’t believe how nice it was.  Wide and open, terrific amenities, very reasonable price.  However, now I’m realizing that the wide open spaces and the huge door is just perfectly sized for a bull to come wandering in and lay waste to my china.

I’m pushing this analogy too far, of course.  End Users are not bulls (most of them, anyway) and IT departments don’t (or surely should not) view their user community with that kind of suspicion.  However, there is this sort of perfect collision taking place already in the the 3.0 platform that I expect will only get worse in SP 2010.  SharePoint already empowers and encourages End Users to define and implement their own solutions.

That’s great and all, but the fact is that it’s still a very technical product and still calls for the kind of vigorous business requirements analysis, design and general planning and management that technical projects require to be successful.  These are not the kind of skills that a lot of End Users have in their bag of tricks, especially when the focus is on a technical product like SharePoint. 

I’ve given this a lot of thought over the last year or so and I don’t see any easy answer.  It really boils down to education and training.  I think that SP 2010 is going to change the game a bit and it’s going to play out differently and in slow motion as companies roll out their SP 2010 solutions over 2010 and beyond.  In order to succeed, End Users will need to transform themselves and get a little IT religion.  They’ll need to learn a little bit about proper requirements
analysis.  They will need some design documentation that clearly identifies business process workflow, for instance.  They need to understand fundamental concepts like CRUD (create, update and delete), dev/test/qa/prod environments and how to use that infrastructure to properly deploy solutions that live a nice long time and bend (not break) in response to changes in an organization.

In the coming weeks, I plan to try and provide some of my own new ideas, as well as link to the great work done by many other authors (on www.endusersharepoint.com and elsewhere) so that interested End Users can learn that old time IT religion.  Keep tuned.

</end>

Subscribe to my blog.

Follow me on Twitter at http://www.twitter.com/pagalvin

[Note: This article cross-posted to End User SharePoint here: http://www.endusersharepoint.com/2009/09/09/sharepoint-a-case-study-in-ask-the-expert/]

Sometimes, when you’re working as a consultant (as a profession, or in a consultative role within your company), you find yourself living in an Onion story.  The Onion has a series of articles called “Ask an [expert] about [some problem]”.  This follows the famous “Dear Abby” format where a concerned person is asking for personal advice.  The onion’s “expert”, however, is so focused on his/her area of expertise and current problems that the expert ignores the question entirely and rambles on about his area of expertise.  As consultants, we need to keep that in mind all the time and avoid falling into that trap.  It’s classically described like this – “when you use a hammer all day long to solve your problems, everything starts to look like a nail.”  We professional consultants are always on guard against that kind of thing, but we come into contact with people who are serious professionals in their own role, but are not consultants.  They don’t have the same need or training to do otherwise.

Last week, I wrote about one of my company’s clients and an on-going project we have to enable high quality collaboration between various eye doctors in the US and Canada performing clinical research on rare disease.  In addition to leveraging core SharePoint features to enable that collaboration, we’re also working an expense submission and approval process.  It’s complicated because we have so many actors:

• A handful of individuals at different doctors’ practices who can enter expenses on line.
  • There are over 40 doctors’ practices.
  • At some practices, the doctor uses the system directly.
  • At many practices, the doctor’s staff uses the system directly.
• A financial administrator (who works for my direct client) who reviews the expenses for accuracy and relevancy, approving or denying them at the organizational level.
• A 3rd party accounts payable group.  These people pay all of the bills for out client, not just bills coming out of the rare disease study. 

The Accounts Payable group has been a challenge.  Working with them yesterday reminded me of the Onion series.  In my role as business consultant, I explained the need to the accounts payable company:

• Clinical studies sites (doctors’ practices) incur study-related expenses.
• They log onto the “web site” and enter their expenses using an online form.  In this case, the “web site” is hosted with SharePoint and the expenses are entered into an InfoPath form.  Expense receipts are scanned, uploaded and attached directly to the form.
• An automated workflow process seeks approval from the appropriate financial administrator.
• You, dear 3rd party AP company – please review and approve or deny this expense.  I’ll send it to you any way that you want (within reason).At this point in the discussion, I don’t really care how it needs to be bundled.  I want to work with the AP group to understand what they need and want.

When I explained the need, the 3rd party took a deep dive into their internal mumbo jumbo lingo about expense approval processes, Oracle codes, vice presidential signatures, 90 day turn-arounds, etc.  And panic.  I shouldn’t forget about the panic.  One of the bed rock requirements of the consulting profession is to learn how to communicate with people like that who are themselves not trained or necessarily feel a need to do the same.  Among other things, it’s one of the best parts of being a consultant.  You get to enter a world populated with business people with completely different perspectives.  I imagine it’s a little bit like entering the mind of a serial killer, except that you aren’t ruined for life after the experience (though entering the mind of an AP manager isn’t a walk in the park 🙂 [see important note below***] ).

One of the great things about our technical world as SharePoint people is that we have ready-made answers to many of the very valid concerns that people such as my AP contact have.  Is it secure?  How do I know that the expense was properly vetted?  Can I, as the final payer, see all the details of the expense?  How do I do that?  What if I look at those details and don’t approve of them?  Can I reject them?  What happens if the organization changes and the original approver is no longer around?  Can we easily change the process to reflect changes in the system? Can I revisit this expense a year later if and when I get audited and need to defend the payment?

As SharePoint people, we can see how to answer those questions.  In my client’s case, we answer them more or less like this:

• InfoPath form to allow sites to record their expenses and submit them for approval.
• Sites can return to the site to view the status of their expense report at any time.
• As significant events occur (e.g. the expense is approved and submitted for payment), the system proactively notifies them by email.
• The system notifies the financial administrator once a report has been submitted for approval.
• Financial administrator approves or denies the request.
• Upon approval, the expense is bundled up into an email and sent to the 3rd party payer organization.
• The 3rd party payer has all the information they need to review the expense and can access the SharePoint environment to dig into the details (primarily audit history to verify the “truth” of the expenses).
• 3rd party payer can approve or reject the payment using their own internal process.  They record that outcome back in the SharePoint site (which triggers an email notification to appropriate people).
• In future, it would be nice to cut out this stilly email process and instead feed the expense information directly into their system.

In conclusion, there’s a life style here that I describe from the professional consultant’s point of view, but which applies almost equally to full time employees in a BA and/or power user role.  Work patiently with the experts in your company and extract the core business requirements as best you can.  With a deep understanding of SharePoint features and functions to draw upon, more often than not, you’ll be able to answer concerns and offer ways to improve everyone’s work day leveraging core SharePoint features.

***Important note: I really don’t mean to compare AP people to serial killers.  However, I could probably name some AP pro’s who have probably wished they could get a restraining order against me stalking them and asking over and over again.  “Where’s my check?”  “Where’s my check?”  “Where’s my check?”

</end>

Subscribe to my blog.

Follow me on Twitter at http://www.twitter.com/pagalvin

[Note: this blog post is cross posted at Mark Miller’s site here: http://www.endusersharepoint.com/?p=1897]

One of my company’s more unusual clients is a New York City doctor who is a leader in his particular field of medicine (eye care).  Like many doctors, he has a strong interest in research.  He wanted to do some research on a rare eye disorder that affects a relatively small number of people in the U.S. and Canada.  I don’t know the number, but it’s really too small for a large pharmaceutical company to invest its own private funds with an eye toward eventual commercial success.  I’m sure large pharma’s do some amount of research into rare diseases, but I believe that the U.S. government is probably the largest source of funding.  Like anything, resources are scarce.  Many doctors across the country want to perform research and trials.  As a result, there’s more than a little competition for that government funding.  This is where my company and SharePoint enter the picture.

The fundamental idea is that a master organization will recruit other doctors across the country and enlist those doctors’ practices in a particular research study.  These individual practices must sign up with the master organization and then, subsequently, sign up for a particular study.  The relationships look like this:

• One master organization.
• Many different doctor’s practices sign up with the master organization.
• The master organization obtains funding for individual studies.  At the outset, there is just the one study on a specific rare eye disease although we’re already ramping up for another study.
• Individual doctors’ practices sign up for specific studies.  A specific practice could sign up for one or multiple studies.

The master organization itself is broken down into groups:

• Executive committee
• Steering committee
• Individual study committees
• Administration
• others

Finally, when a specific doctor’s practice signs up to participate in a study, they need to provide professionals to fulfill a variety of roles:

• Investigators (including a primary investigator, normally a doctor, along with one or more additional investigators)
• Coordinators
• Technicians
• Grants administrators
• others

The above roles have very specific and highly proscribed roles that vary by study.  I won’t get into more detail here, but if you’re interested, leave a comment or email me.

And now I can answer the question, SharePoint – What’s it good for?  The answer – it’s really good for this scenario.

This intro is already longer than I expected, so I’ll summarize the vital role that SharePoint plays in the solution and dive into details in a future article (if you can’t wait, email me or leave a comment and I’ll be happy to discuss and maybe even try to do a demo).  We are leveraging a wide array of SharePoint features to support this concept:

• Sites for committees, individual roles (coordinator sites, investigator sites, etc). 
• Security to make sure that different practices don’t see other practices’ data.
• InfoPath forms services for online form entry.  This is a particularly big win.  Normally, these difficult forms are printed, mailed to the practices, filled out and mailed back.  The advantages to the online forms are obvious.  They do introduce some complexities (licensing and human) but that’s another story.
• Out of the box web parts, like announcements (when does committee [x] meet?) and meeting work spaces.
• Forms based authentication in combination with a CodePlex tool to provide self-registration and password forget features.
• Customized lists and list views for visibility into study activities which simply aren’t possible with pure paper and pencil approaches.

With the exception of the forms based authentication module and a handful of InfoPath forms, this project is using nearly all out of the box SharePoint functionality.

Before I wrap up this min-case study, I want to point out something very important – no on involved with this project (aside from my company of course) has any idea that a thing called “SharePoint” is playing such a fundamental technical role.  Nearly all of my end users view this as “the web site.”  Our client values us because we’re solving their business problem.  SharePoint is a great technical blob of goodness, but done right, that’s irrelevant to end users.  They need a problem solved, not a wonderful blob of technology.

</end>

Subscribe to my blog.

Follow me on Twitter at http://www.twitter.com/pagalvin

Or is it just me?  I’ve received three or four calls at my house since late September looking for SharePoint work.  I’m used to the email solicitations, but these phone calls are a little unnerving.  I haven’t had an updated resume on a job site I(like Monster  pr Dice) since almost two years ago exactly.  And back then, my resume was all about BizTalk and MS CRM.  That’s the only place my phone number appears on line anywhere, so far as I know.

</end>

Subscribe to my blog.

Follow me on Twitter at http://www.twitter.com/pagalvin

The closing thought on this otherwise dull article speaks well to problems we often face in the technical community:

"Such dreary developments, anticipated with certainty, must be borne philosophically."

This puts me in mind of one of the presentations I gave at the SharePoint Best Practices conference last month.  I was describing how to get "great" business requirements and someone in the audience asked, in effect, what to do if circumstances are such that it’s impossible to get great requirements.  For example, a given company’s culture places IT in front of the requirements gatherer / business analyst, preventing direct communication with end users.  This is a serious impediment to obtaining great business requirements.  My answer was "walk away."  I’m not a big humorist, so I was surprised at how funny this was to the audience.  However, I’m serious about this.  If you can’t get good requirements, you can be certain that a dreary outcome will result.  Who wants that?  I’m a consultant, so it’s more realistic (although terribly painful and drastic) for me to walk away.  However, if you’re entrenched in a company and don’t want to, or can’t, walk away, George (for once 🙂 ) shows the way.

</end>

Subscribe to my blog.

Follow me on Twitter at http://www.twitter.com/pagalvin

How often does this happen to you?  I’m sitting at my laptop, reading blogs, responding to forum postings, 2 copies of visual studio open and VPN’d out to another server with its own visual studio + 15 browser windows (a typical day) and someone named Samantha (my wife, apparently) tells me, "We have be there in 30 minutes.  Get dressed."

I get up in a daze, wander around the house confusedly, get in a car and next thing I know, I’m at a party with a beer in my hand and someone asks me, "So, what do you do for a living?"

These conversations never go well.

Me: "Ahh … I’m a solutions architect for EMC."

Nameless Person: blank stare

Me: "I work with a product called SharePoint … it’s from Microsoft."

NP: "Aha! I’ve heard of that company!  What is SharePoint?"

Me: "Umm … it does collaboration … people use it to share information … It’s a platform for building busines sol…"

NP: Eyes glazing.

Me: "I’m a programmer."

NP: "Aha!  I know people in my company that do programming!  When I was in high school, I played around with BASIC."

And with that part of the conversation over, we turn to something easier to talk about, like politics.

Anyone care to describe how they handle this? 

</end>

Subscribe to my blog.

I’ve been in classes these past two weeks and one thing that strikes me is that there are a lot of thoughtful, smart people working on SharePoint (as consultants or IT staff) who don’t blog, twitter, seem aware of public message boards like MSDN forum or SharePoint University, maintain Facebook or LinkedIn profiles, etc.  They are pure information consumers.  Not bad, just interesting.

</end>

Subscribe to my blog.

A bunch of years ago, my boss asked me to train some users on a product called Results.  Results is an end user reporting tool.  It’s roughly analogous to SQL Server Reporting Service or Crystal.  At the time, it was designed to run on green tubes (e.g. Wyse 50 terminal) connected to a Unix box via telnet. 

My default answer to any question that starts with "Can you … " is "Yes" and that’s where all the trouble started.

The client was a chemical company out in southern California and had just about wrapped up a major ERP implementation based on QAD’s MFG/PRO.  The implementation plan now called for training power end users on the Results product.

I wasn’t a big user of this tool and had certainly never trained anyone before.  However, I had conducted a number of other training classes and was quick on my feet, so I was not too worried.  Dennis, the real full-time Results instructor, had given me his training material.  Looking back on it now, it’s really quite absurd.  I didn’t know the product well, had never been formally trained on it and had certainly never taught it.  What business did I have training anyone on it? 

To complicate things logistically, I was asked to go and meet someone in Chicago as part of a pre-sales engagement along the way.  The plan was to fly out of New Jersey, go to Chicago, meet for an hour with prospect and then continue on to California. 

Well, I got to Chicago and the sales guy on my team had made some mistake and never confirmed the meeting.  So, I showed up and the prospect wasn’t there.  Awesome.  I pack up and leave and continue on to CA.  Somewhere during this process, I find out that the client is learning less than 24 hours before my arrival that "Paul Galvin" is teaching the class, not Dennis.  The client loves Dennis.  They want to know "who is this Paul Galvin person?"  "Why should we trust him?"  "Why should we pay for him?"  Dennis obviously didn’t subscribe to my "give bad news early" philosophy.  Awesome.

I arrive at the airport and for some incredibly stupid reason, I had checked my luggage.  I made it to LAX but my luggage did not.  For me, losing luggage is a lot like going through the seven stages of grief.  Eventually I make it to the hotel, with no luggage, tired, hungry and wearing my (by now, very crumpled) business suit.  It takes a long time to travel from Newark — to O’Hare — to a client — back to O’Hare — and finally to LAX.

I finally find myself sitting in the hotel room, munching on a snickers bar, exhausted and trying to drum up the energy to scan through the training material again so that I won’t look like a complete ass in front of the class.   This was a bit of a low point for me at the time.

I woke up the next day, did my best to smooth out my suit so that I didn’t look like Willy Loman on a bad day and headed on over to the client.  As is so often the case, in person she was nice, polite and very pleasant.  This stood in stark contrast to her extremely angry emails/voicemails from the previous day.  She leads me about 3 miles through building after building to a sectioned off area in a giant chemical warehouse where we will conduct the class for the next three days.  The 15 or 20 students slowly assemble, most them still expecting Dennis. 

I always start off my training classes by introducing myself, giving some background and writing my contact information on the white board.  As I’m saying, "Good morning, my name is Paul Galvin", I write my name, email and phone number up on the white board in big letters so that everyone can see it clearly.  I address the fact that I’m replacing Dennis and I assure them that I am a suitable replacement, etc. I have everyone briefly tell me their name and what they want to achieve out of the class so that I can tailor things to their specific requirements as I go along.  The usual stuff.

We wrap that up and fire up the projector.  I go to erase my contact info and … I had written it in permanent marker.   I was so embarrassed.  In my mind’s eye, it looked like this: There is this "Paul Galvin" person, last minute replacement for our beloved Dennis.  He’s wearing a crumpled up business suit and unshaven.  He has just written his name huge letters on our white board in permanent marker.  What a sight! 

It all ended happily, however.  This was a chemical company, after all.  A grizzled veteran employee pulled something off the shelf and, probably in violation of EPA regulations, cleared the board.  I managed to stay 1/2 day ahead of the class throughout the course and they gave me a good review in the end.  This cemented my "pinch hitter" reputation at my company.  My luggage arrived the first day, so I was much more presentable days two and three.

As I was taking the red eye back home, I was contemplating "lessons learned".  There was plenty to contemplate.  Communication is key.   Tell clients about changes in plan.  Don’t ever check your luggage at the airport if you can possibly avoid it.  Bring spare "stuff" in case you do check your luggage and it doens’t make it.  I think the most important lesson I learned, however, was this: always test a marker in the lower left-hand corner of a white board before writing, in huge letters, "Paul Galvin".

</end>

Due to some oddball United Airlines flights I took in the mid 90’s, I somehow ended up with an offer to transform "unused miles" into about a dozen free magazine subscriptions.  That is how I ended up subscribing to Scientific American magazine.

As software / consulting people, we encounter many difficult business requirements in our career.  Most the time, we love meeting those requirements and in fact, it’s probably why we think this career is the best in the world.  I occasionally wonder just what in the world would I have done with myself if I had been born at any other time in history.  How terrible would it be to miss out on the kinds of work I get to do now, at this time and place in world history?  I think: pretty terrible.

Over the years, some of the requirements I’ve faced have been extremely challenging to meet.  Complex SharePoint stuff, building web processing frameworks based on non-web-friendly technology, complex BizTalk orchestrations and the like.  We can all (hopefully) look proudly back on our career and say, "yeah, that was a hard one to solve, but in the end I pwned that sumbitch!"  Better yet, even more interesting and fun challenges await.

I personally think that my resume, in this respect, is pretty deep and I’m pretty proud of it (though I know my wife will never understand 1/20th of it).  But this week, I was reading an article about the Large Hadron Collider in my Scientific American magazine and had one of those rare humbling moments where I realized that despite my "giant" status in certain circles or how deep I think my well of experience, there are real giants in completely different worlds. 

The people on the LHC team have some really thorny issues to manage.  Consider the Moon.  I don’t really think much about the Moon (though I’ve been very suspicious about it since I learned it’s slowing the Earth’s rotation, which can’t be a good thing for us Humans in the long term).  But, the LHC team does have to worry.  LHC’s measuring devices are so sensitive that they are affected by the Moon’s (Earth-rotation-slowing-and-eventually-killing-all-life) gravity.  That’s a heck of a requirement to meet — produce correct measurements despite the Moon’s interference.

I was pondering that issue when I read this sentence: "The first level will receive and analyze data from only a subset of all the detector’s components, from which it can pick out promising events based on isolated factors such as whether an energetic muon was spotted flying out at a large angle from the beam axis."  Really … ?  I don’t play in that kind of sandbox and never will.

Next time I’m out with some friends, I’m going to raise a toast to the good people working on the LHC, hope they don’t successfully weigh the Higgs boson particle and curse the Moon.  I suggest you do the same.  It will be quite the toast 🙂

</end>