Inspired: How to Create Tech Products Customers Will Love

Lessons from Top Companies

• It doesn’t matter how good your engineering team is if they are not building something worthwhile
• There is a tremendous difference between how the best companies produce products and how most companies produce products
  • In other words, the state of the art is not the same as the state of practice

Behind Every Great Product

• There is a product manager who is responsible for tying technology, design and business principles to ensure that the solution solves a problem that users have
• Often requires a lot of work (we’re talking around 60 hrs/week)

Technology-Powered Products and Services

• These include consumer-service products, such as e-commerce sites or marketplaces (e.g., Netflix, Airbnb, or Etsy), social media (e.g., Facebook, LinkedIn, or Twitter), business services (e.g., Salesforce.com, Workday, or Workiva), consumer devices (e.g., Apple, Sonos, or Tesla), and mobile applications (e.g., Uber, Audible, or Instagram)
• They do not need to be purely digital and can be blends of offline and online experiences

Startups: Getting to Product/Market Fit

• 3 stages of companies: startups, growth-stage and enterprise companies
• Startup: new product company that has yet to achieve product/market fit.
• Product manager role is covered by one of the founders
• You’re in a race to achieve product/market fit before money runs out
• The few companies that do really well move fast and learn from their mistakes. Furthermore, they often are good at product discovery

Growth-stage Companies: Scaling to Success

• You have to hire more people and firgure out how to replicate earlier successes with adjacent products/services
• Very challenging: product teams don’t understand their impact, technical debt rises, sales and marketing cant use their old tactics, leadership styles often don’t scale

Enterprise Companies: Consistent Product Innovation

• Strong tech companies know that they need to constantly create new value for their consumers and markets.
  • This isn’t just about tweaking products; its about developing products so that they reach their full potential
• The problem is that there are many stakeholders to this business who are trying to protect it. This often means shutting down new ventures or ideas, which is a signal of a death spiral
• Product teams complain about a lack of vision, empowerment and bureacracy, while leaders are frustrated with the slow pace of innovation

Root Causes of Failed Product Efforts

• Typical product development effort: idea → bizcase → roadmap → requirements → design → build → test → deploy
  • Agile methodology only enters in the build phase, so its still a waterfall process
• Top 10 reasons why products fail *
  1. Sales-driven or stakeholder-driven ideas are not the best source of ideas. Lack of team empowerment
  2. Business cases are useless because there’s no way of knowing the project cost or its value
  3. Half of the ideas on the roadmap are not going to work. Preplanning ideas is a horrendous proposition. Secondly, many of the ideas requires several iterations until it reaches value
  4. The above process gives more of a project manager role to a product manager. They are just gathering requirements and writing specs
  5. Design is basically ‘lipstick of the pig’ in this process because they had very little input in how the product was being made
  6. Engineers are usually the best source of innovation , but they are put in at the very end of the process
  7. Agile methodology is entered too late so you miss out on many of the benefits
  8. Project-centric, which leads to projects just as output. Products are not output; they are outcome
  9. Customer validation happens way too late
  10. Huge opportunity cost

Beyond Lean and Agile

• Lean and Agile are not silver-bullet methodologies: they won’t fix everything.
• Successful product teams use the following three overarching principles: *
  1. Risks are tackled up front, rather than at the end: risks tackled prior to building, including value risk (will customers buy it), usability risk (will users figure out how to use it), feasibility risk (can engineers build it with resources), and business viability risk (does this work for various aspect of business)
  2. Products are defined and designed collaboratively by everyone
  3. It’s about solving the problem, not implementing features:

Key Concepts

Holistic Product

• Includes functionality, technology, UX design, monetization, how users and customers are attracted and can include offline experiences

Continuous Discovery and Delivery

• Two essential activities in all product team: need to discover product to be built and deliver product to market
• Product manager and the designer are trying to discover the right solution, while the engineers are delivering

Product Discovery

• This is where we are tackling the various risks before building anything
• Purpose is to separate the good ideas from the bad. Output is a validated product backlog
• Need to find answers to the following questions: *
  1. Will the user buy this or choose to use it?
  2. Can the user figure out how to use this?
  3. Can our engineers build this?
  4. Can our stakeholders support this?

Prototypes

• Product discovery involves a lot of quick experiment. To do this quickly and inexpensively, product teams use prototypes
  • Prototypes come in various forms, but they will never require the full time and effort of a product
• Strong teams test product ideas on the order of 10-20 per week

Product Delivery

• Purpose of prototypes and experiments is to quickly come upwith something that provides evidence that product is worth building and delivering to customer
• Delivery requires product to be at necessary scale and performance and that the product works as advertised

Products and Product/Market Fit

• End goal is for product/market fit: smallest possible product that nmeets the needs of a specific market of customers
• This is the result of the delivery phases of the product

Product Vision

• Refers to the longer-term objectiver of the product, usually 2-10 years out
• Prototypes for rapid experiments → products to achieve product/market fit → deliver company’s product vision
• Remember, MVP’s are not minimum viable products, but minimum viable prototypes

The Right People

• It’s all about the product team. We need to optimize for the effectiveness of product teams

Principles of Strong Teams

• Product teams is a group of people who bring together different specialized skills and responsibilities and feel ownership for a product
• Despite the different products, product teams have some underlying similarities

Team of Missionaries

• Product teams should be missionaries, not mercenaries. They should be building products because they are true believers in the vision
• Should feel like you are part of a startup in a larger company

Team Composition

• Typical product team si comprised of product manager, product designer, and between 2-10 engineers
• Designer not needed if it’s not a user-facing experience

Team Empowerment and Accountability

• They are there to solve hard problems for businesses. Given clear objectives and deliver on those objectives
• Empowered to figure out best way to meet those objectives and are accountable for results

Team Size

• You should keep a product team in the 2-pizza rule
• What is more important than size is that there is a balance of skills

Team Reporting Structure

• It is an intentionally flat organization, so there are no people managers
• Usually, each person reports to their functional manager, i.e. product refers to head of product, design talks to head of design, engineering talks to engineering manager

Team Collaboration

• The only way for the product to come alive is for everyone to help each other out and work out solutions together

Team Location

• Usually, teams are sitting right next to each other so that they can easily see each other screens
• Co-located teams perform substantially better than dispersed teams

Team Scope

• Two dimensions: type of work (should be anything and everything for a product) and scope of work (could be whole product or a small part of customer experience)
• Many ways to divide up work for different teams (crucial as startup grows): different types of users, different devices, different customer journeys, architecture
• No correct way to slice up work, but engineering and product management should be working in lockstep

Team Duration

• Important to keep team so that relationship dynamics build and they can work more effectively with each other
• Need to keep people long enough to develop expertise
• Remember, this is a product team and not a project team. Shouldn’t be disbanded right after product is launched!

Team Autonomy

• Allow team to solve problems in best way that they think of, should not be imposed top-down
• Should try to minimize dependency between teams

The Product Manager

• Taking issues and major decisions to the CEO or letting stakeholders come to a consensus over a product decision is NOT how product managers should work
• The product manager is responsible for evaluating opportunities and determining what gets built and delivered to customers
  • Product backlog: what needs to be done
• We hold the product manager accountable for the success of the product
• Deep knowledge of the customer: you should know their issues, their desires, how they think, how they work and how they decide to buy.
  • Requires qualitative and quantitative learning
  • Should be an expert of product as well
• Deep knowledge of data: need to be comfortable with analytics (sales, usage, A/B tests)
• Deep knowledge of business: you should be comfortable explaining every single part of your business, how it works and what role your product plays
  • Know your various stakeholders and their constraints
  • Success requires you to convince each stakeholder that you understand their constraints and that you are delivering products consistent with their constraints
• Deep knowledge of market/industry: competitors , key trends, customer behaviour, following industry analysts, role of social media
  • Your product needs to be substantially better than your competitors in order to seize the market
  • Product should be well-incoporated with the existing ecosystem
• It takes almost 2 months to get fully up to speed with the product
• Traits of people that thrive in this role:
  • Smart: curious, learns quickly, using new tech or new business models
  • Creative: think outside normal product box to solve a business problem
  • Persistent: push companies beyond comfort zone with data and persistent communication. Be able to build bridges across different functions in the face of stubbornness
• You will not get a good work-life balance in product management
• Steps to becoming a great product manager:
  • Deeply understand your user base and become the go-to person when it comes to anything about users
  • Establish good relationships with key stakeholders
  • Become an undisputed expert in the product and industry
  • Work hard to build a strong and collaborative culture within your product team
• Product owner is simply an Agile role that is responsible for the product backlog. The product manager should be the product owner, but product owner is only a small subset of the PM’s responsibilities
• Two academic courses that every PM should take: intro to programming and intro to business finance

The Product Designer

• Product designers are responsible for:
  • Product discovery: should be a partner with product manager in creating new solutions
  • Holistic user experience: need to make sure that touchpoints with users are smooth as possible
  • Prototyping
  • User Testing: shouldn’t just be when the prototype/product is ready, but should be continuous to glean insights
  • Interaction and visual design:
• Without designers, the following could happen:
  • You have no design background and do the design work yourself
  • Leave engineers to design the product
  • You do the interaction design and leave the visual design to the designer
• The above never provide good results. You need a trained product designer to differentiate yourself from the competitors (eg. Facebook and Google have put in enormous effort to get product designers)
• The designer should be sitting right next to you to fully understand the product.
  • They should be there with the product manager at every single moment of user/stakeholder contact
  • Give designer as much room as possible to think of design solutions themselves
  • Allow and encourage iteration

The Engineers

• The most important relationship you can foster as a product manager is one with the engineers
• Engineers need to know everything about the user but they don’t need you to spell out how to do their job
• You need to make sure that at least one engineer is avaliable for product discovery tasks

Product Marketing Managers

• Usually spread out among different product teams
• They are responsible for bringing the market to the product, such as positioning, messaging and go-to-marketing strategies
• If you don’t have a product marketing manager, responsibility lies on the product manager.

The Role of Leadership

• Leadership traditionally means recruiting, developing and retaining exceptional talent, but leaders in product orgs need to have a holistic view of the product
• You need leaders in product management (VP of product, principal product manager) that can identify and review conflicts
• You need leaders in product design, who can make sure that the design is roughly the same throughout the entire company’s suite of products
• You need leaders in engineering who is reviewing the general architecture of software implementation and has a strategy on dealing with technical debt
• Make sure that each of these people has a second-in-command in case something happens or they leave the company

Head of Product

• Manages the product designers and managers. Very difficult role but has high potential for impact

• Key competencies: team development, product vision, execution, product culture

  • Team development: recruiting, training and retaining talent is a top priority. They should be not only good at developing products, but they are also good at developing people
  • Product vision: product leader should be complimenting the CEO with vision
  • Execution: should be great planner, great at discovery and development and can scale

Head of Technology

• Remember, relationship with your engineers is the most crucial relationship you need to develop as a PM
• Major role is to use technology strategically to accomplish the business goals of the organization
• 6 major responsibilities:
  • Organization: needs to build a strong management that strives to help engineers grow professionally
  • Leadership: works with other key stakeholders to represent technology’s role in major business decisions
  • Delivery: makes sure that organization can rapidly, reliably and repeatedly ship great products to the market
  • Architecture: self-explanatory
  • Discovery: make sure that the engineers are involved in product discovery
  • Evangelism: serves as the company spokesperson on all things technology

Delivery Manager

• Special type of project manager that removes obstacles for team. Also have the role of Scrum master

Principles of Structuring Product Teams

• One of the most difficult decisions to make is how to split up company product among dozens of product teams
• Key principles for structuring product teams
  • Alignment with investment strategy: make sure that the product teams are structured in a way that reflects what the company is more/less invested in
  • Minimize dependencies: eliminate the dependencies between product teams
  • Ownership and autonomy: self-explanatory
  • Maximize leverage: create teams that are developing shared services that are used by many other product teams
  • Product vision and strategy: ensure you have structured teams that are capable of delivering this product vision
  • Team size: each product team should have only 1 product manager and between 2-10 engineers
  • Alignment with architecture: want to make sure that the talent of the product team is well-suited for the archiecture needed
  • Alignment with user: product teams should focus on different types of users
  • Alignment with business: could align with business units
  • Structure is a moving target: review your team structure every few months to make sure that they fall in line with the above principles

The Right Product

Product Roadmaps

• Remember that product teams should be focusing on outcome over output. Roadmaps are all about output
• Roadmap: prioritized list of features that a product team needs to work on over a quarter, year or more
  • Sometimes comes from the top from stakeholders (stakeholder-driven roadmap) or from product manager
  • Includes features, projects and big project intiatives with deadlines and time frames. Doesn’t include smaller fixes, bugs or optimizations

Problems with Product Roadmaps

• Remember the inconvenient truths about product: half of the ideas are not going to pan out and the ideas that do pan out require several iterations
• Weak teams simply plod through the roadmaps. When they come across a needed iteration, they blame stakeholders and then try to squeeze another iteration in the product roadmap
• Strong teams iterate much quicker (create prototypes in days, not weeks)
• Crux of the problem is that people view the roadmaps as commitment, which doesn’t provide the freedom to iterate for better solutions
• Sometimes we do need to commit to a delivery for a certain date, but this is avoided and is termed a high-integrity commitment

Roadmap Alternative

• Why have roadmaps even existed for so long if they continue to give product teams a massive headache?
  • Management wants to make sure that product teams are working on high-priority items
  • Businesses need date-based commitments
• Product teams are the best people to figure out how to solve business problems, but they need:
  • Product vision and strategy: the big picture for what the organization is trying to accomplish and the plan to get there
  • Business objectives: should be specific and prioritized
• Management should be providing product teams with the business objectives and the key metrics to measure success
• High-integrity commitments are used for date-based commitments
  • Don’t commit to dates before you have even started product discovery. Ask for time for discovery of a necessary solution to business problem and then start plotting dates
• Why is this method superior? Team is more motivated (missionary vs mercenary), team is tackling an important problem for the business, and model is able to embrace likelihood of the two inconvenient truths
• Roadmaps can be changed to reflect this. Instead of having specific features or projects, have business objectives instead

Product Vision and Strategy

• Product vision describes the future of the product 2-5 years out (hardware companies is usually longer)
  • Persuasive piece that could be in the form of a storyboard, a simple narrative or a visiontype
  • You want to inspire stakeholders and teams to realize your vision
• Product strategy is the sequences of products and releases that will realize the product vision
  • Should be focused on different product/market fits, such as user types, geography, market verticals, or a key sequence
  • Focus on a single market at a time and obsess over them
• The organization needs to set product vision and strategy, not individual product teams. However, they should be aware of the vision and strategy and it should be inspiring for all
• How to prioritize markets:
  • 3 inputs: total addressable market, time to market, go to market (sales channel)
  • Balance out these 3 inputs and create product strategy from there

Principles of Product Vision

• 10 key principles for product vision
• 1. Start with why: use the product vision to articulate purpose
  2. Fall in love with the problem, not the solution
  3. Don’t be afraid to think big: it should be big enough to inspire everyone
  4. Don’t be afraid to disrupt yourself: create new value for your customers, otherwise someone else will
  5. Product vision needs to inspire: create something that is exciting and focuses on users
  6. Determine trends: use the trends to help solve customer problems
  7. Skate to where the puck is heading, not where it was: identify change
  8. Be stubborn on vision but flexible on details
  9. Realize that any product vision is leap of faith
  10. Evangelize continously and relentlessly

Principles of Product Strategy

• Many different ways of approaching strategy, but good ones share the common 5 principles
• 1. Focus on one target market/persona at a time: don’t try to please everyone with a single release. It will at least be loved by some
  2. Product strategy must be aligned with business strategy
  3. Product strategy must be aligned with sales and go-to-market strategy
  4. Obsess over customers not competitors: customers only leave companies if the company stops taking care of them
  5. Communicate strategy across organization

Product Principles

• Product principles speak to the nature of the products that you want to create
• Provide guidelines on creating products. Clear statement of what you believe in

Product Objectives

• Objectives and key results (OKR) framework has been essential in guiding product progress. It is based on two principles:
  • Summarized by Patton’s quote: “Never tell people how to do things. Tell them what to do, and they will surprise you with their ingenuity.”
  • Only release features/products that solves underlying business problems

The OKR Technique

• Important points when using OKR:
  • Objectives should be qualitative while key results are quantitative/measurable
  • Key results should be a measure of business results, not product output
  • Objectives should be about the organization and product team, not functional or personal objectives
  • Find good time interval (usually annual for organization objectives and quarterly for product team)
  • Keep the number of OKRs small (1-3 objectives, each with 1-3 key results)
  • Product teams should track active progress against objectives on a weekly basis
  • Objective should cover what the team needs to accomplish, not every small thing
  • Team should feel accountable for achieving objectives. If they fail, it is worth a good post-mortem or retro
  • Agree as an organization how you will be scoring key results (usually on a 0-1 scale, 0 being no progress, 0.3 is bare minimum, 0.7 is you did more than minimum, 1 is exceptional results)
  • Establish when a key result is a high-integrity commitment, where success is binary (you either delivered or you didn’t)
  • Be transparent about the objectives that your team is working on
  • Senior management should be determining OKR on a organization wide basis, products teams should be determining OKR for a team basis

Product Team Objectives

• OKRs help teams align with company objectives and helps teams understand what their role is in the company
• OKRs should not be done on a functional basis, as this leads to confusion in a product team.

Product Objectives @ Scale

• With larger organizations, product teams need help in defining their OKRs. Specifically, management needs to decide which teams are better suited for tackling certain objectives
• Don’t forget about the support teams! They need their own objectives as well
• Each product team needs to propose key results and management should review them and determine what help they need to achieve those key results
• Management should help connect the dots between different teams’ OKRs
• High-integrity commitments become a bigger problem for scaling companies so ensure that you are actively tracking them
• Some enterprise companies have different business units, which should have their own OKRs as well

Product Evangelism

• Helps people imagine the future and inspiring teams to create that future
• Especially important for startup CEOs and PMs
• How to sell the dream: *
  1. Use a prototype
  2. Share the pain: show the engineers and designers what customers need by bringing them along to customer visits
  3. Share the vision: have a clear understanding of product vision, strategy and principles
  4. Share learnings generously: especially after customer feedback
  5. Share credit generously
  6. Learn how to give a great demo: remember your strategy should be different based on the audience
  7. Do your homework: be the undisputed expert on users and markets
  8. Be genuinely excited
  9. Learn to show enthusiasm
  10. Spend time with your team

The Right Process

Product Discovery

• Two very signficant problems that need to be tackled by teams:
  • Discovering in detail what the customer solution might be
  • Need to make sure that solution works for many users, not a series of specials (need to test out many ideas, quickly and inexpensively)
• Discovering products requires fast and quick experimentation, but delivering great products requires time and effort from engineering

Principles of Product Discovery

• Purpose of product discovery is to address 4 risks: value, usability, feasibility and business viability
• Principles: *
  1. We cannot count on customers to tell us what to build: they don’t know what is possible. All product teams are responsible for is making a solution that solves a problem
  2. The most important thing is to establish compelling value
  3. Coming up with good user experience is harder and delivers more value
  4. Functionality, design and technology are intertwined: this is why you need design, business and tech working in tandem
  5. We expect many ideas will fail. The ones that do will require several iterations
  6. Validate ideas on real users and customers
  7. Validate ideas in the fastest and cheapest way possible
  8. Validate during discovery, not after
  9. Validate business viability during discovery, not after
  10. Share learnings
• Consider ethical side of products too: should we build it
• Iterations in discovery is simply trying a new idea. Strong product teams have 10-20 iterations/week!

Overview of Product Discovery Techniques

• Framing techniques: helps us to quickly identify issues that needs to be tackled in product discovery
  • Clarify the problem being solved, tease out risks, figure out where to spend time
• Planning techniques: can help with identifying bigger challenges and planning around them
• Ideation techniques: designed to provide team with a wealth of promising solutions
• Prototyping techniques
• Testing techniques: all about discovery solutions that address the 4 risks. Only validate what you need to validate and pick right technique for that solution
  • Testing feasibility: designed for engineers to identify areas of concern (new tech, scale, third-party components)
  • Testing usability: designed for designers to address area of concern (complex workflows, interaction design, confusion sources)
  • Testing value: need to ensure users will buy/continue to use the product
  • Testing business viability: can afford any costs, able to be sold by sales team, great for stakeholders, sticks to brand
• Transformation techniques: used to help organizations to transform way of work
• Some of these techniques are quantitative, other are qualititative. Others are proofs, others are evidence.

Discovery Framing Techniques

• Big projects need to ensure that their discovery work is aligned and deals with appropriate risk. Thus, there are two goals: *
  1. Whole team is on same page and are clear on business problem, user problems and objectives
  2. Identify big risks that need to be tackled in discovery process
• Don’t only focus on technology and usability risk. Value risk and business risk are arguably more important and tricky to solve
• If there are no risks, then proceed to delivery. No need to be conservative and test out all hypotheses. Use validation to iron out risks
• Remember to fall in love with the problem, not the solution. More often than not, initial solutions will not solve the problem, so you need to have some motivation to keep going
• Opportunity assesment technique:
  • Answer four key questions: what business objective does this address, how will you know that this succeeded, what problem does this solve for customers, what type of customers are we focused on
  • Business objective: tie it back to team’s assigned objectives
  • Key results: should map to one of the KRs in the OKRs of the team
  • Customer problem: although our products are meant to help the company, they should ultimately help users
  • Target market: helps product teams narrow in on particular user segment
  • Other questions might need to be answered in this framework
• Customer letter technique:
  • Start the process with a pretend product press release
  • This helps the PM to structure work backwards and intended to keep focus on users
  • You can change this to a letter from a imagined, happy end user to the CEO
• Startup canvas technique:
• • Used especially when you are tackling a new business opportunity, like startups
  • Startup canvases are similar to business model canvas and help highlight risks that you need to tackle
  • Many startups are facing value risk. They need their product to be substantially and demonstrably better than existing solutions

Discovery Planning Techniques

• Story map technique:
  • Arrange the user flow on the horizontal dimension and the stories vertically for each step of the flow, in order of most critical to least
  • You can determine where to literally draw the line for meeting objectives
  • Can use it to frame prototypes and update according to how people interact with product
  • Based on all of this, pretty easy to move into backlog
• Customer discovery program technique:
  • Reference customer: real customer who runs product in production, paid for the product and is willing to tell others about the product
  • This technique develops a set of reference customers without actually discovering or developing the product
  • Used mainly for new products/new markets
  • For businesses you want to have at least 6 reference customers from the same vertical to instill confidence in prospective customers that the product is worth it
  • These reference customers need to truly understand the problem and hope for a solution that fits exactly their needs; not technologists
  • They need to have people and time to work closely with the product team
  • You are building a product for everyone, not them. However, your product should ultimately work extremely well for reference customers
  • Should be a single solution that works for all reference customers, not a hodgepodge of different features
  • If you’re overwhelmed with users who want to be reference customers, that means this is an important problem. If you are struggling to find people, then that indicates that the deman for a solution for this problem is not there
  • Product/market fit can be measure by how easily it was to get your references customers

Discovery Ideation Techniques

• Customer interview:
  • Extremely powerful technique with several different ways going about it
  • You need to understand the following: are your customers really who you think you are, do they really have the problems you think they have, how does the customer solve the problem today, what would make them switch?
  • Frequency: have at least 2-3 hours of interviews per week
  • Purpose: you’re not proving anything, you’re discovering
  • Recruiting users and customers: looking for people in target market, only 1 hour of their time
  • Location: best to see them in their local habitat
  • Preparation: think about what problem you think they have and how you will confirm it
  • Who should attend: PM takes notes, designer drives, engineers watches
  • Interview: natural, informal, open-ended
  • Afterward: debrief and make sure everyone got same results
  • Can also test out product ideas with user
• Concierge test:
  • Do the customer’s job for them, manually and personally
  • Gives you a deeper view on what customers need
• Customer misbehaviour
  • Encouraging users to use products in a non-intended way is quite powerful and can lead to new ideas
  • Lean in and figure out what problem the user is trying to solve
  • Many companies use this strategy for public APIs, and they quickly glean product opportunities
• Hack Days:
  • Give full freedom for people to work on undirected or directed problems
  • Also creates a team bonding experience

Discover Prototyping Techniques

• Different types of prototypes: feasibility prototypes for engineers, user prototypes (low-fidelity to high-fidelity), live-data prototypes (can GET/POST real data)
• Principles of prototypes: *
  1. Should not require nearly as much time and effort as a product
  2. Forces you to think about idea at a deeper level
  3. Powerful tool for collaboration
  4. Recognize need to change fidelity when needed
  5. Needs to tackle a product list for discovery purpose, but can be used as a prototype spec
• Feasibility technique:
  • Engineer writes throwaway code to test a concept
  • Should be relatively short unless exploring a new technology
  • This is important because you don’t want a product that is too technically complex for current resources
• User prototype technique:
  • Can range from a variety of fidelities. Built mainly by product designers
  • This is not appropriate for determining the value of the product
• Live-data prototype technique:
  • Used when you actually need to collect some user data
  • This is not an OK substitute for an actual product
• Hybrid prototype technique:
  • Wizard of Oz prototype: front-end of high-fidelity prototype with someone manually running backend
• Many of these are not scalable but they give us excellent ideas on products

Discovery Testing Techniques

• Four things we look for: value, business viability, engineering feasbility, usability
• Address value and usability first, then feasbility and viability
• Usability testing:
  • Try to get user researchers to do this because they are trained
  • Recruit users: customer-discovery program, advertise, e-mail customers, go to where users congregate, compensate
  • Preparing for the test: use a high-fidelity prototype, get PM & designer & engineer, define set of tasks to test, assign roles (driver, note-taker, debrief), determine where (remote is fine but not meant for value testing)
  • Testing: be clear that you’re testing a prototype and not the interviewee, ask them what they think about front-page and glean first-visitor info, users should be in use mode and not critique mode, keep quiet, avoid giving help, parrot, pay attention to body language
  • Summary: write up a short summary email of key findings ine testing session
• Value testing:
  • Features not only have to match, but they have to be substantially better than existing solutions
  • Need to test for demand
    • Fake door demand test: put a button and see how many people click thorugh when you advertise your fake product that you haven’t actually built
    • Landing page demand test: same thing but you dedicate a whole site
  • Qualitatively: look at the user comments and see whether they are willing to pay for it
    • Single most important technique for product discovery
    • Use a user interview and then proceed to value-ask questions
    • Ask user if they would be willing to pay for it, how likely they would recommend this to people, would they be willing to schedule time to set up this system, ask for credentials of product they would be switching from
    • Iterate from your interviews and try to understand the discrepancies by doing more tests
  • Quantitatively: need to create some sort of metric to measure how well product solves the problem that users want solved
    • This is where the live-data prototype comes into play
    • A/B test where 99% of people are shown the existing product and 1% shown the live-data prototype
    • Invite-only testing or use users in the customer-discovery program
    • Role of analytics:
      • Gain a better understanding of users and customers
      • Measure product progress
      • Prove whether a product is working
      • Inform product decisions
      • Inspire product work
      • Analytics explains what is happening, it won’t explain why. That’s why qualitative tests are needed
  • For risk averse companies: you need to innovate or you will die. If risk is something you are wary of, just use simple A/B tests that only 1-2% of your customers will be exposed to or invite-only tests
• Feasibility testing:
  • Give engineers some time to make estimates and decisions on technology. If they came along with user tests, they know what users want and don’t want and have spent time thinking about it
  • Sometimes they will need to make a feasbility prototype
• Business viability testing:
  • Show marketing and sales your prototypes before you build anything to ensure that they can market the product well
  • Understand customer success model (low touch vs high touch) and talk to leaders
  • Sit down with finance to model the costs of your product
  • Legal is incredibly important so make sure to sit down with them to ensure legality
  • Product demo is selling the product, walkthorugh is a thorough analysis and walking through of the product to show that they havent missed anything. Very different from user tests

Discovery Transformation Techniques:

• Product discovery sprint (from the book Sprint)
• Pilot team technique: roll out the change to a limited subset of the organization. Kind of like an A/B test for internal
• Weaning off roadmaps: make it very clear that you are addressing a particular business objective when presented with the roadmap. Tie all results/successes/failures back to the business objective and eventually team will move away from roadmap

Process @ Scale

• Managing stakeholders:
  • A simple test to determine whether someone is a stakeholder or not is to see if they have veto power on your product
  • PMs need to understand the constraints and considerations of each stakeholder and build trust that you will address them in a product
  • Be open and transparent with stakeholders and have 1:1 with them, about half hour per stakeholder
  • Show your solutions in discovery phase, not delivery phase
  • Move discussions from opinions to data by running tests
  • Do not trust a sign-off on anything asides from a high-fidelity prototype
• Communicating product learnings
  • Communicate learnings every couple of weeks in a all-company hands-on meeting
  • VP of Product should be doing this and should be very high-level
  • Helps product teams to stay focused on the mission and helps other employees to understand what product teams are doing

The Right Culture

Good Product Team/Bad Product Team

• Good product teams pursue a product vision with a missionary-like zeal; bad product teams are mercenaries
• Good product teams generate ideas from customer problems, business objectives, and new technology; bad teams get them from sales and customers
• Good teams understand the stakeholders and their constraints which are considered in product formulation; bad teams gather requirements from stakeholders
• Good teams use techniques to rapidly test many ideas; bad teams hold meetings to determine roadmaps
• Good teams have eng, design and PM sit side by side while bad teams allow them to be siloed
• Good teams are constantly trying new ideas and protecting brand; bad teams are still waiting for permission to test
• Good teams have all the skillsets; bad teams don’t even know what skillsets they need
• Good teams engage engineers in prototype testing; bad teams only show prototypes to engineers for estimates
• Good teams engage with end users and customers every week; bad teams think they are the customer
• Good teams know that their favourite ideas won’t work out or will need a lot of iterations; bad teams are happy to build what’s on the product roadmap
• Good teams are fast; bad teams complain about slowness due to people unable to complete work
• Good teams always instrument their work to understand how their product is used; bad teams consider analytics as a nice-to-have
• Good teams integrate and release continuously; bad teams have an extremely painful integration phase and then a massive release
• Good teams obsess over reference customers. Bad teams obsess over competitors
• Good teams celebrate impact on business objectives, bad teams celebrate releases

Top Reasons for Loss of Innovation

• Organizations that lose innovation are missing some of the following:
  • Customer-centric organization
  • Compelling product vision
  • Focused product strategy
  • Strong product managers
  • Stable product teams
  • Engineers in discovery
  • Corporate courage
  • Empowered product teams
  • Product mindset
  • Time to innovate

Top Reasons for a Loss of Velocity

• Reasons may include:
  • Technical debt
  • Lack of strong product managers
  • Lack of delivery management
  • Infrequent release cycles
  • Lack of product vision and strategy
  • Lack of co-located, durable teams
  • Not including engineers in product ideation
  • Not utilizing product design in discovery
  • Changing priorities
  • Consensus culture