Eating Hours

James Bourke recently wrote what I found to be an extremely thoughtful list of Ten Questions You Should Ask a Cloud Service Provider over on the CPA2BIZ newsletter site. We still detest this term “cloud” in the case of “Cloud Service Provider”, but given that we certainly qualify under this definition as it was intended here, I thought that providing concrete answers to these 10 questions may help take some of the mystery out of our “cloud” infrastructure.

To directly quote Jim, these questions are:

Some of the top questions that I would recommend asking before making your final decision (in no specific order, as order of importance will vary depending on the type of data and application deployed)

Here are his questions and our answers in the context of each of the products we develop and maintain (currently including workpapers.com, auditconfiramtions.com, and sas70registry.com but would also be applicable to any other products we develop in the future as we are a big believer in this technology stack):

Where will my data be stored?

Our applications are built on the Amazon Web Services computing platform. We selected Amazon for a large variety reasons that I will discuss throughout these questions but not the least of which was because they have both a very thorough SAS 70 report done by Ernst & Young and an excellent track record of providing the best security, availability, and scalability among hosting companies available. Specifically, all of the data stored in our applications is physically stored in MySQL databases. Each application’s production database runs on a dedicated EC2 database server instance. Each EC2 instance is replicated across Amazon’s computing facilities in a given “availability zone”. We are currently using the Eastern United States availability zones meaning production data is geographically located in Amazon run data centers in Virginia and New Jersey (possibly others as Amazon adds facilities in the East).

The only exception to this is the actual documents and files uploaded and attached to procedures in workpapers.com. These documents are stored in a dedicated production storage bucket on Amazon’s S3 service. Storing these documents to S3 rather than our server instances gives the files “durability” of 99.999999% which basically means that three geographically separated Amazon data centers would all have to suffer catastrophic disaster for us to be at risk of losing even one file. Additionally, it makes our storage pool virtually limitless. We never have to upgrade hardware storage which helps us keep pricing reasonable. What this means geographically is that these files are replicated to several physical Amazon data center locations that span the United States. We are also adding additional S3 storage in Ireland that will serve our clients in Europe, the Middle East, and Africa giving them better performance than we can provide from our storage pool in the US.

What type of security and controls are in place to protect confidential and sensitive client data?

We use a careful combination of both preventive and detective physical and logical access controls to protect sensitive client data. Physical access to computing facilities is managed by our hosting provider, Amazon Web Services, whose detailed SAS 70 report we have reviewed thoroughly to verify those controls are adequate. It is particularly important to remember that not only are the facilities physically secured, but that the servers we use are virtualized so even an individual who is inside the data center would not be able to specifically identify our servers and execute an attack. The Amazon SAS 70 also covers logical access controls to host operating systems (e.g. the operating systems on the physical servers that, in turn, host our virtual machines). From there, iTickmark controls security of the virtual machines ourselves directly as our systems administrators are the only ones with access to the operating systems and databases on those virtual servers. Our own SAS 70 report (which we expect to be issued in October of 2010) will cover the testing of those controls. As you can see, there is an extremely thorough “defense in depth” strategy in place here that would be essentially impossible for our customers to replicate in their own, onsite environment. Even still, we implement detective and audit controls to further test systems security. For instance, McAfee Secure scans our systems and certifies their security on a daily basis, 365 days/year.

What type of redundancy does the vendor have in place?

Our most important redundancy controls is the real-time data replication that we get from using a virtualized systems infrastructure (Amazon EC2) and storage pool (Amazon S3). Unlike most vendors in the accounting and audit space who operate on older, internal infrastructure, every piece of data we have is instantly replicated across multiple, geographically separated, data centers. That makes data stored with us ultra-durable compared to other vendors or using in-house systems.

Even despite this real-time redundancy, we do our own backups of both our production application databases and the mass of files and documents stored in workpapers.com. We take twice-daily backups of each database, then store the backup to S3 so that it is as safe, redundant, etc. as all of our other files. With files and documents, we copy them to a second S3 storage bucket in case the ultra-unlikely event occurs where Amazon does lose three data centers at one time. We have never needed to restore a file from this second line of backups or the effective “backups of the backups” that are created but they are there, should we ever need to.

Finally, we also maintain a relationship with an entirely separate hosting provider should everything from Amazon across the globe fail, even though that seems nearly impossible. We have thoroughly tested the procedure to restore our applications to this second provider (Rackspace) and are able to move our infrastructure, applications, and customer data over there within just a couple of hours.

What is the vendor’s data retention policy?

Our data retention policy with regards to customer data is to let the customer control it. For instance, in workpapers.com we keep client engagements that have been archived there permanently. However, if the customer deletes an engagement because it is older than their data retention policy requires, we warn the customer then delete it immediately. All of the backup data we take (as noted above) is retained for a maximum of one year. We do not keep backup data longer than one year expressly for the purpose of not having customer data they thought was gone available in the event of litigation.

Who will have ownership of that data?

100% of customer data, without exception, is owned by that customer. We do not and will not ever allow anyone to mine customer data for any reason. This is a contractual obligation on our end as we provide these terms in our terms of service and privacy policy for each product.

In what type of format will my data be stored?

We find data lock-in and format trapping the most annoying and egregious practice among both traditional and “cloud” software vendors today. Your data comes out of our applications as easily as it went in. The most important of our products in this regard is of course workpapers.com. Everything you enter in the system itself like audit procedures, work plans, findings, testing results, etc. is available for download in both Excel and PDF formats. Additionally, you can of course download all the files and documents you have uploaded into the system back out at anytime. If you want to move to a new software platform, we understand… we change vendors sometimes also. You can certainly take every document with you. AuditConfirmations is the same way. Download all of your completed confirmations in a PDF format and move on to another vendor or back to using paper for confirmation procedures. No problem.

What happens in the event of data loss or corruption?

We have not ever lost any customer data, easy as that. In fact, we have never utilized our second line backups outlined above. Every potential data integrity concern we have ever seen is corrected in real-time by our use of virtualization and replication.

What happens in the event of loss of data? Who is responsible?

iTickmark maintains specialized insurance coverage that specifically covers the risks associated with being a hosted application provider that covers data breach and loss for all of our customers. However, we cannot afford to lose our own data in our applications so we realize our customers can’t either. Thus, we consider insurance only a resolution tool in the worst possible scenario. We are happy to provide evidence of our insurance coverage to customers requiring that.

What if you end up in a fee dispute or disagreement with the vendor?

We have never had a fee dispute because we don’t trap (or even ask) customers to engage in long term contracts. We hate trying to buy a cell phone contract for example so why would we put our customers through something similar? Our products are operated under a very simple “pay for what use, stop whenever you want” model with open, public pricing available. We guarantee 99.9% uptime on all our products and if we miss that for any reason other than a planned upgrade which we have notified customers about ahead of time, we’ll be happy to refund that month’s service. Easy as that.

How financially stable is the vendor and who or what is behind their primary funding source?

We are a privately owned software company and have been in business since January of 2008. We are funded through the operations of our business and we have been profitable since the inception of the business. As our business and our customers grow, our applications and computing infrastructure grow with it. This is the same way most audit firms themselves grow and finance their operations. We frankly feel we are CONSIDERABLY more stable under this operational model than many software companies that are bought and sold between large vendors who may or may not kill off the products you use and care about or venture-backed companies that may have their funding cut-off anytime for whatever reason and don’t the profits to sustain operations. We finance our products, we grow our business slowly over time, and we use our products ourselves. We have as much skin in the game as any of our customers.

I (Dan Zitting) personally serve as the CEO of iTickmark. I am also a partner in the public accounting firm Linford & Company LLP which provides two services: SAS 70 audits and royalty/licensing audits. iTickmark’s products were born because they were what we needed in our work and were not available in the market nor was an equivalent available in a price range reasonable for our small practice. It is our background in public accounting that makes our products great. They are built directly from the practitioner’s perspective rather than a software developer’s. It is also why we are so careful with customer data, my own firm is one of those customers. I am the only person with involvement in both of these businesses and using my position as the CEO of iTickmark to identify customers whose clients Linford & Company could pursue would be a gross violation of ethics rules. If it is a concern, just give me a call directly and we can make arrangements to relieve it.

New Rule

If we are to continue using formal RFPs as a valid tool for business, they must start resembling a clear, concise, and genuine attempt to find the best service provider rather than the Publisher’s Clearinghouse Sweepstakes (where no one you know ever seems to actually win).

We met some great people… that is the upside. The brilliant, hard working people who are in CPA firms, in the trenches, getting stuff done. That was the upside. People I can relate too, people who don’t understand Twitter but understand they can’t have their tax return software down for 9 hours in the middle of busy season. THOSE are the people that make the accounting profession great. The people my 57 year old father (who owns a small town, but several million dollar in revenue hardware store) uses to get his business’ taxes done every year, for the past 30 years. Sometimes though, I get the sense they are an endangered species.

Here at his conference we had the “Hardware Store CPAs” attending sessions, and more importantly several keynote presentations, that focused almost exclusively on things like “innovation”, “adding value”, and “social media”. Kind of a bunch of shit honestly. I can tell you, my father pays his CPA real money (she does value-based billing… hint, hint) every year because she makes his life better, every single quarter at payment time. The years when a big tax issue comes up (a co-op stock revaluation a few years back for example), she hammers him with serious value-based bills and he just smiles and signs the check because HE didn’t have to incur any brain damage dealing with it. Instead, he just got to go on peddling nuts and bolts as usual. No matter how many “added value tweets” you make or “recommendations on your Facebook fan page” you have, you are not getting my old man’s business, sorry. He loves his accountant, who knows his business backwards and forwards, far too much.

I bring all of this up just to say, lets get real. We need GOOD accountants. More of them (many, many more of them). Accountants who work in small firms, bill a fair amount for the work they do, and help make owning a business manageable. Accountants who frankly spend less time setting up Facebook fan pages and more time ticking and tying. Social media, web 2.0 tools, and all this stuff is great and certainly can be valuable communication tools in certain settings… lets just not let the profession lose sight of what is most important. It is our job as a vendor also to encourage this by focusing our efforts on building real-world enablers while ignoring the constant urge to create needless “innovation”.

With that all out of the way, I offer my “conference winner” (the best thing) and “conference loser” (the worst thing) about the 2010 AICPA TECH+/Practitioner’s Symposium Conference.

CONFERENCE WINNER:

Practioner/Technology Combination - Combining the conferences was probably a byproduct of the poor economy and registration expectations related to it. However, what a great idea it was. I thoroughly enjoyed being able to engage in a nice mix of sessions and meet such a wide variety of people. I hope it continues as such next year.

CONFERENCE LOSER:

Mingle Stick – Frankly, a poor idea. Business cards work brilliantly for exchanging contact information, BUT they (much more importantly) also express the values of a company through design, type, material, etc. The Mingle Stick is not only a bit cheesy, but it kills this important form of expression.

New Rule:

Vendors have to eliminate the term “cloud computing” from their marketing vocabulary unless they actually operate datacenters in some sort of aircraft or high in the Himalayan mountains.

After a lot of hard work by our crack development and design team, we are really excited to launch our newest service, The SAS 70 Registry. Like the Audit Fuel execution framework and Workpapers.com, The SAS 70 Registry is a service we developed directly to address the needs I saw developing at my own CPA firm. We are still regularly asked if there is a SAS 70 “logo” or “certification seal” that service organizations can put on their website to market the fact they have completed a SAS 70 audit. The other thing I see all the time are press releases announcing a service organization has completed a SAS 70 audit, however nearly all of them use technically inaccurate or somewhat misleading language. So, as you look around you see that getting marketing leverage from a company’s SAS 70 is difficult and disjointed at best, but SAS 70 is a lot of work and a big investment that should help a company acquire new customers. SAS 70 (and SSAE 16 to follow) just needs a more clear, yet technically accurate brand in the marketplace.

Enter sas70registry.com. What we have done is developed a set of marketing tools for use by service organizations whom we have verified have successfully completed a SAS 70 audit. Included with each registration are “Click to Verify” web seals, a set of beautiful but simple high resolution marketing graphics that can easily be integrated with most any marketing theme, and a press release template for SAS 70 completion announcements. At the same time, for those service organizations that opt-in, we create a search-optimized public directory of those companies which we have verified so potential customers (user organizations) have a great place to start looking for SAS 70 qualified service providers. We are also extending an offer to any CPA firm that specializes in doing SAS 70 audits to use the SAS 70 Registry materials in their marketing efforts if they would like to.

We think this service will really help enhance the perception of SAS 70 in the marketplace as well as ease the transition confusion among non-auditors over to SSAE 16. Right now we are “soft launching” as we sign up a few organizations to begin filling out the directory. More will come as we develop the brand over the next few months.

As always thanks for the support and we look forward to using this service to enhance SAS 70 for everyone in the SAS 70 community. If you have questions, please reach out. We love hearing from you!

There are a lot of articles and posts flying around right now (and always) about studying for the CPA exam, selecting a review program, etc. We’ve all been there and the exam is certainly intimidating when you’re just getting started on the process, but there is a simple answer.

The profession certainly needs better accountants. We need more accountants who really understand the fundamental assumptions, principles, and constraints of GAAP; who effectively leverage clear critical thinking to appropriately interpret industry guidance and solve operational problems; and who can write (and otherwise communicate) effectively. However, I would contend that these critical basics are not (in general) what is tested by the exam. The exam is, in large part, a simple exercise in memorization and basic reasoning, just like any test. All the review courses know that and that is why they use the cramming methods they do. Keep this in mind…

It is a totally logical contention that 75 is a PERFECT score on the CPA exam. Why? Because if you scored exactly 75, it means you spent the absolute minimum time studying you possibly could have and still became a CPA. A 100 gets you nothing but maybe your name in an extra email blast as some big firm welcomes you, like everyone else, into staff 1 servitude.

So, rather than waste hours of time and thousands of dollars in review courses where you rehash intermediate accounting class from undergrad in a lecture setting, I suggest a more “real” approach. The exam is composed of questions. You can buy tons of practice questions on CD or via download for under a hundred bucks. Skip the course and go straight to answering practice questions. Start with 100, answer every one of them. Then see how many you missed. Don’t be discouraged even if you only got even 25% right because the key is step 2. You MUST now review every question you missed. Reread the question, then review your answer and the correct answer. It is well established in cognitive science that you retain knowledge better if you have tried something cold once, failed, and then see the correct answer rather than just being told correct material from the start. Now, redo the ones you missed then review the ones you missed a second time. Once you are to the point of having correct answers on 85 out of the 100, rinse and repeat with a new set of 100.

If you go through the process above for a few hundred questions in each section and do 5 or so practice simulations for each section that requires simulations, you will PASS (but not score perfect on) the exam, you will be a $1000+ wealthier, and you won’t have had to schedule around weeks of Saturday review courses.

This line of reasoning brings up another question entirely however and that is… “But doesn’t the profession want people who really understand the fundamental assumptions, principles, and constraints of GAAP; who effectively leverage clear critical thinking to appropriately interpret industry guidance and solve operational problems; and who can write (and otherwise communicate) effectively… not just people who beat the test?” and the answer to that question is certainly yes, but no test that can be finished in a matter of 10-15 hours will ever ensure that. It ALWAYS simply too easy to beat a test.

When people don’t pay you for your time, they rarely value it.

via @sh on Twitter

RFPs are a complete waste of time. Worse yet, are the proposals generally submitted in professional services in response to the RFPs. This business practice dinosaur needs to be stopped, here is why.

Requests for Proposal (RFPs) (Summary: People who prepare RFPs are wasting their time and probably need to be reassigned to value creating activities)

In preparing an RFP, a company typically compiles page after page of background information, detailed requirements, desired vendor qualifications, awards received, and a near endless supply of other (frankly bullshit) metrics that have effectively no correlation with the proposing firms’ ability to execute the engagement effectively, on time (or early), within budget, and with valuable results. In fact, we find that the actual reason the overwhelming majority of RFPs are prepared is for the benefit of the preparer. That person is typically in some sort of compliance or procurement role, whether audit related, IT related, etc. Generally speaking, executive management does not consider people in these roles to be extraordinarily valuable (nor should they). Therefore, this person prepares RFPs with extensive, unnecessary questioning in order to make himself/herself look “smart” and to make the compliance/audit process look complicated (and thus their role important).

The reality, in the case of audit-related services at least, is much more simple. The organization faces a set of risks… possibly compliance related, fraud related, financial related, etc. The organization simply needs a person or team who can quickly, effectively, and concretely identify those risks and can execute audit and/or other evaluation procedures to determine those risks have been mitigated. It probably should take one paragraph to solicit a bid on that, not a 40 page document.

Proposals (Summary: People who prepare RFP responses are wasting their time and probably need to be reassigned to value creating activities)

Proposals produced in response to RFPs are considerably MORE painful to read even than the RFPs themselves. This is the reason I have trouble believing that anyone of significant importance is a proponent of executing an RFP process (unless it wastes someone else’s time, not their own). Writing and reviewing proposals are both busy work. At best, the “Round One” evaluation consists of reviewing proposals for what is, effectively, a set of buzzwords or meaningless key phrases. Once the RFPs are thinned out, the RFP manager then selects a subset of the proposing firms to present to management. It is of course from there that the final selection is made. Yet everyone in the industry knows the undercurrent of any RFP process is that the secretly pre-determined preferred provider of executive management (you know the one, the provider that the CFO specifically asked the RFP manager to include in the RFP process because he plays golf with a certain partner over there). Somehow, no matter how bad the proposal is, the “secretly preferred provider” ALWAYS manages to make it through to the interview/presentation round. Oddly enough, they manage to win more-often-than-not.

Stop the Madness (Summary: Respect service providers and potential vendors)

Lets cut the BS. Everyone knows that the highest level person involved on the client side will ultimately decide who wins the work. Therefore, why waste everyone’s time in the interim? This practice of using RFPs to solicit bids and leverage those to force selection of the preferred provider is disrespectful to those involved, a gross waste of time at best, and a flat out attempt at stealing intellectual property and unethical price leveraging at worst.

A Better Way (Summary: Review “real” work and consider firm price quotes to quickly select the best vendor)

Instead of wasting company time preparing the RFP and accumulating a listing of totally empty and meaningless “qualifications”, focus on what matters:

1. Which vendors the client actually knows (and potentially trusts) already
2. What the service provider has actually done
3. Where the client and service provider can meet on pricing

For example, with the clients that hire us for technical audit services… the client usually knows and trusts us, we have extensive examples of our work that are from “real” audits and fully sanitized to remove confidential information, and we offer fair pricing that requires (and leaves room for) no negotiation. In a one page document or 30 second discussion, it is clear whether our services are a fit or not. No time wasted on either side.

Firm Policy Outcomes

This is what we recommend for firms trying to perpetuate a positive market and equitable bid environment:

1. Proposals are often solicited in an effort to leverage pricing against current provider in an RFP process where the client has no intention of switching. This is a classic lose/lose situation. Prevent inter-industry lose/lose games by not proposing on RFPs where the current provider is asked to bid.
2. Prepare “real”, concrete work examples that exhibit prior, successful achievement of the client’s goal.
3. Decide what it will take to do the job, bid that, and don’t back off of it. Be fair, do not scale up the bid price purely because you think the client can or will pay it.

Following this process, we can eliminate lengthy RFP processes that waste time on both sides, avoid those horrible meetings where vendors drag you through a deck of 40 powerpoint slides, and perpetuate the hiring of providers who can illustrate true skill and demonstrated ability while at the same time avoiding the constant price degradation imposed by under qualified providers.

This post on the blog “re: The Auditors” is the best read on the auditors’ role in the financial crisis that I have seen, highly recommended.

To quote a bit:

Gatekeepers? Or foxes in the hen house?

The auditor’s role is to be a gatekeeper. A watchdog. An advocate for shareholders. This is their public duty.

This public trust is subsidized by a government-sponsored franchise. All companies listed on major stock exchanges must have an audit opinion. Audit firms are meant to be shareholders’ first line of defense, and they are hired by and report to the independent Audit Committee of the Board of Directors.

And yet the same audit firms that stood by and watched Bear Stearns and Lehman Brothers fail – Deloitte and Ernst &Young – are recipients of lucrative government contracts to audit or monitor the taxpayers’ investment in the bailed out firms. Deloitte, the Bear Stearns and Merrill Lynch auditor, works for the US Federal Reserve system. Ernst & Young, Lehman’s auditor, is working for the US Treasury on the original $700 billion TARP program and with the Fed on the AIG bailout.

The whole situation is dramatically overcomplicated by those involved, and I do think it would be fair to argue that this is the case because those involved like it that way. It makes clear problems murky and hard to assign blame for. There is definitely a fundamental problem at work when a huge chunk of the large institutions on Wall Street are all suddenly so broke they need a tax payer bailout but at the same time ALL have a clean audit opinion. Simple question, simple answer… a fundamentally flawed system (and party interests mis-aligned with party responsibilities).

Great work Francine, simplifying the situation to expose the underlying problem.

It seems to me that hourly rates are a dinosaur that should die just about any time now. There is a fundamental problem with charging for anything by the hour… Nothing of value is ever actually accomplished in an hour, so how can you charge that way and truly expect value delivered to directly correlate with price paid?

Furthermore, everyone in professional services knows how frustrating it can be to lose work (whether tax, audit, consulting, legal, etc) to a competitor because of a difference in hourly rate, when there is no doubt that the competing firm is not as specialized in the particular service area. we’ve all been in a spot where there’s no doubt a competitor will take more time to deliver an inferior work product. resulting in the client ultimately paying more for less. The client can’t be blamed however because all firms promise “efficiency”, how would they know if one firm will be much cheaper, despite a higher hourly rate?

We all know the typical flow of events with accountants, lawyers, and other “consultants”. It goes something like this:

1. Consultant gives client a rate per hour and establishes a “budgeted” number of hours (lets use 400 hours (10 weeks) as an example).
2. Client selects consultant because they are the cheapest rate per hour.
3. Project begins and everything is reported “on target” for 75-80% of the engagement (first 7-8 weeks) despite the fact that in reality not a heck of a lot is actually being delivered.
4. Middle of week nine, the job’s not done.
5. End of week 12, management begins asking why things aren’t done and getting expensive.
6. Consultant explains what happened and why it is not their fault the project is running over budget.
7. In status meetings it is always reported the project is “95% done”, yet it never actually seems to finish.
8. Around 20 weeks of time billed, final deliverables are handed over.
9. Client gets the final invoice, realizes budget was doubled, and is not happy.
10. Consultant gets paid and uses the client as a “case study” in how they deliver value when selling to the next potential client, despite knowing client was not happy.

This scenario happens on more projects than not, I would argue, and it is not healthy. Yet it could probably be avoided if the client had selected a specialized consultant whose “bread and butter” was the particular project type the client was doing in the first place. It could have further been avoided if there were clear, short-term goals that needed to be accomplished every two weeks rather than focusing only on the end goal and wasting too much time on unnecessary planning and “re-planning” when things changed.

I contend a new model for service delivery is required. Rather than offer an hourly rate, offer an “iteration rate”. No one can predict how long a project will take when it is a project that will take multiple people more than a month to finish, so why try? Explain that to the client and instead tell them you will accomplish simple two week goals. Each two week period will be considered an “iteration” and simple, straight forward expectations for each iteration will be established the first Monday morning of each iteration, and a quick debrief to determine if the expectations were met conducted the last Friday afternoon of the iteration. No extensive planning periods are necessary as nothing goes according to plans for a full 10 weeks anyway. There will be established price for each iteration.

This will give the client a chance to focus on whether value is being immediately delivered. If the first few iterations aren’t working, find and fix the problem or fire the consultant. Easy. Even easier, most service firms aren’t sufficiently confident in their delivery to offer their services on this “do or die” basis in the first place, which should speak volumes about their actual abilities.

Besides the obvious benefits to the client, there are definite benefits to the service firm as well. First, it would not be constantly fighting the rate per hour battle with every new opportunity, yet it still won’t have to take on the risk of pricing very large (thus inherently unpredictable) projects on an overall project basis. Second, there is a nice scheduling benefit as resources can be scheduled in iterations, making arranging blocks of time between staff and clients much easier. Finally, engagement executive time can be used much more effectively because meeting iteration-based expectations is much easier than trying to manage to overall expectations for large, unpredictable projects.

Obviously this would take a significant culture shift for both the service provider and client but there is no doubt agreement on doing so would lead to a better relationship, more concrete and real-time communications, and a more productive combined team. In our firm, we are generally pricing this way, taking smaller projects and pricing them on a project basis, and it is working great. We certainly do better on value delivered per hour on some jobs compared to others, but we quickly identify those differences and learn from them. As further testament, in delivering software development, we do everything on an iteration basis, NEVER focusing on anything beyond one or two iterations out. It has worked brilliantly there as well.

Hopefully more firms in a wide variety service industries will join in and also help kill off the rate per hour model. Even better, hopefully clients start demanding it so there is increased transparency in which firms deliver high-performance and which don’t.