Chapter 15: Product Use Testing, AB testing, Ab test, AB Testing, SaaS Key Terms, Saas Terminology, Market Sizing, SAAS metrics, Rob's Decode & Conquer +, Decode and Conquer, Product Metric Frameworks, Interview - Product design, Interview - Product...

A versus B

- A and B are are identical except for one variation that might affect a user's behavior - Dependent variable: conversion rate (e.g., number of order placement/traffic) - Identical to a between-subjects design

explore first

- Alternatives A/B have an equal chance of selection - Use hypothesis testing to select a 'Winner'

AB testing takeaways

- Do a power calculation first to estimate sample size - Do not stop the test early - Use a valid hypothesis - don't use a scattergun approach - Perform a second 'validation' test -Bayesian experiment design (You can stop your experiment at any time and make perfectly valid inference)

Marketing Ramp-Up Phase

"I think we got it" then begin work on packaging, branding, advertising, etc.

ab testing vs bandit

- More efficient: they move traffic towards winning variations gradually, instead of forcing you to wait for a "final answer" at the end of an experiment - Faster: samples that would have gone to obviously inferior variations can be assigned to potential winners

what do you do in a multi armed bandit

- Randomly assign customers to be assigned to A or B (or more alternatives) - Based on the response, increase the probability of assigning to the alternative with higher response

multi armed bandit

- Standard A/B testing : first learn, then earn - Multi-armed bandit problems: simultaneously "learn and earn" (or "explore and exploit")

AB testing cons

- easy to get wrong - most winning AB test results are illusory

type II error

- false negative - probability of not seeing a positive results when the null hypothesis is not true

type I error

- false positive - probability of a positive result when the null hypothesis is true

precision

- proportion of true positives in the positive results - It's a function of power, significance level and prevalence ex: if you find 10 positive results, if 9 of them are true (not from the Type 1 error (a)), the precision is 90%

statistical power

- the probability that a statistical test correctly rejects the null hypothesis when it is a false - power = 1- B - usually 80-90% - if power is low, you miss true effects

Arguments for product use testing

1. You want to deliver quality products 2. Risks and costs are small compared to the loss of earnings flow from a successful product

repeated significance tests

1. power calculation to get desirable sample size 2. Run an A/B test until the sample size is achieved 3. Check the results

A hypoechoic mass is seen in the renal pelvis of the kidney. Which of the following is the most likely diagnosis? A. TCC B. Angiomyolipoma C. Adenoma D. Hypernephroma E. Renal lymphoma

A

A nonencapsulated collection of necrotic and edematous peripancreatic tissues is termed A. Phlegmon B. Pseudocyst C. Pseudoaneurysm D. Ascites E. Cystadenoma

A

A patient has been referred to US to rule out the presence of pancreatic cancer. The cancer is most likely to be located in which part of the pancreas? A. Head B. Neck C. Uncinate process D. Body E. Tail

A

All of the following are indications for US evaluation of the retroperitoneum except A. to rule out a splenic abscess B. to follow up a hematoma in the quadratus lumborum muscles C. an unexplained decrease in hematocrit D. to rule out a hematoma in Gerota's fascia E. following a renal transplant

A

An elderly man falls on a wooden chair and strikes his left side. He catches the fall with both hands, but has to struggle to get back to a standing position. Although in pain, he does not report his injury. One day later he falls again and is unable to sit up. Paramedics arrive to find the patient pale and clammy with a blood pressure of 60/40. At the hospital, a sonogram is ordered. The sonographer should be careful to search for A. splenic rupture B. prostate obstruction C. GB rupture D. an abscess E. a renal artery aneurysm

A

During insonation of the pancreas, you notice a prominent vessel just posterior to the pancreatic neck. What vessel are you imaging? A. Portal-splenic confluence B. IMV C. Left renal vein D. SMA E. GDA

A

Part of your routine protocol for pancreatic imaging is to comment on the echogenicity of the organ. What is the normal echogenicity of the normal pancreas? A. Either isoechoic or hyperechoic compared to the liver B. Always hypoechoic compared to the liver C. Always hyperechoic compared to the liver D. Always isoechoic compared to the liver E. The pancreatic echogenicity should never be compared to the liver

A

The accessory pancreatic duct, which is sometimes visible sonographically, is known as A. Duct of Santorinin B. Cystic duct C. Duct of Oddi D. Duct of Vater E. Duct of Wirsung

A

The most sensitive lab test for determining renal dysfunction is A. serum creatinine B. BUN C. 23 hour urine collection D. specific gravity E. AU

A

What is the anatomic relationship of the CBD to the pancreas? A. The CBD is posterior to the head of the pancreas B. The CBD is anterior to the head of the pancreas C. The CBD is medial to the head of the pancreas D. The CBD is superior to the head of the pancreas E. The CBD is lateral to the tail of the pancreas

A

What is the most common cause of acute pancreatitis? A. Obstruction of the pancreatic duct by biliary calculi B. Pancreatic divisum C. Alcohol abuse D. Trauma E. Crohn's disease

A

What structure can you use to identify the posterior aspect of the head of the pancreas? A. CBD B. CHA C. GDA D. Splenic vein E. Left renal vein

A

Which of the following is a true cyst? A. parenchymal renal cyst B. Biloma C. Parapelvic cyst D. Pancreatic pseudocyst E. Seroma

A

Which part of the pancreas does the duodenum encircle? A. Head B. Body C. Tail D. Neck E. All of the above

A

Which part of the pancreas generally has the largest dimensions? A. Head B. Neck C. Body D. Tail E. Uncinate process

A

Your are scanning a patient with increased pancreatic enzymes and WBC count. Which of the following conditions is most commonly associated with these lab findings? A. Acute pancreatitis B. Chronic pancreatitis C. Adenocarcinoma of the pancreas D. Islet cell tumor E. Metastasis to the pancreas

A

Extrinsic causes of hydronephrosis include all of the following except A. a calculus B. uterine fibroids C. trauma D. pregnancy E. ovarian neoplasms

A ( calculus is intrinsic, all the others are extrinsic)

Which of the following cells are responsible for the endocrine function of the pancreas? A. Islet cells B. T cells C. Islets of Langerhans D. Acinar cells E. RBC

A (Islet cells secrete insulin and are found in the Islets of Langerhans, the site from which endocrine tumors originate)

Sonographic depiction of acute renal vein thrombosis includes all of the following except A. renal vein dilation distal to the occlusion B. an enlarged, hypoechoic kidney C. low level echoes in the renal vein D. absent or reversed diastolic flow in the intrarenal arteries E. decreased or no flow on Doppler study

A (There will be low level echoes occasionally seen in the main renal vein. A high resistance arterial waveform in the renal artery with no venous flow detected on Doppler. There will not be renal vein dilation distal to the thrombus because of lack of flow. The kidney will be enlarged.(

AB testing

A randomized experiment with two variants: A and B

A fatal tumor that involves the exocrine portion of the pancreas and predominately occurs in elderly male smokers is A. an islet cell tumor B. adenocarcinoma C. cystadenocarcinoma D. a pseudocyst E. phlegmon

B

All of the following are sonographic indications of rejection in a transplanted kidney except A. increased parenchymal echogenicity B. bladder distension C. high resistance Doppler pattern D. abnormal decrease in renal size E. presence of hydrohephrosis

B

During insonation of the pancreas, you routinely image a vessel coursing anterior to the uncinate process. What is this vessel? A. IMV B. SMA C. GDA D. Left renal vein E. Splenic artery

B

In a trans view of the pancreas what vessel is seen posterior to the SMA? A. IVC B. ab aorta C. Right renal artery D. IMV E. Splenic vein

B

Renal sinus lipomatosis A. is another term for a renal cyst B. causes an increase in echogenicity and size of the renal sinus C. is a term for anechoic areas secondary to obesity D. results in increased cortical echoes E. is a form of angiolyolipoma

B

The cells of the intestinal mucosa secrete a hormone that causes the GB to contract and release bile. This hormone is A. lecithin B. cholecystokinin C. cholesterol D. bilirubin E. renin

B

To exclude a false positive diagnosis of hydronephrosis, the sonographer should A. change to a lower frequency transducer B. perform prevoid and postvoid images of the kidneys C. defer to CT scan D. repeat the scan in 24-48 hours E. defer to MRI

B

Two days after a renal transplant, the most common cause of acute renal failure is A. cyclosporine toxicity B. acute tubular necrosis C. renal obstruction D. acute pyelonephritis E. acute rejection

B

What is the most common US appearance of pancreatic adenocarcinoma? A. Hyperechoic mass B. Hypoechoic mass C. Cystic mass D. Calcified mass E. Complex mass with both cystic and solid components

B

What is the relationship of the SMA to the pancreas? A. Posterior to the tail B. Posterior to the neck C. Superior to the body D. Cephalad to the head E. Lateral to the tail

B

Which of the following is an endocrine tumor? A. Adenocarcinoma B. Islet cell tumor C. Cystadenocarcinoma D. Lymphagioma E. Pancreaticoblastoma

B

Which vessel is located at the superior border of the pancreas? A. SMA B. Celiac trunk C. Left renal vein D. Right renal artery E. IMA

B

You are performing an US exam on a patient with a history of repeated bouts of pancreatitis. What would you most likely use color Doppler for in this study? A. Evaluate for increased flow in the pancreatic parenchyma B. Improve detectability of possible pseudoaneurysms C. Look for flow direction in the SMA D. Assess the quality of flow in the abdominal aorta E. Rule out thrombus in the IVC

B

You are scanning a 52 year old male with a history of alcohol abuse. US findings include a hyperechoic mass in the head of the pancreas, dilation of the pancreatic and CBD, and diffuse calcification within the pancreas. Which of the following conditions is most likely present? A. Acute pancreatitis B. Chronic pancreatitis C. Adenocarcinoma D. Cystadenocarcinoma E. Islet cell tumor

B

You are scanning a patient with a history of alcohol abuse and liver cirrhosis. The pancreatic tissue is heterogeneous. Calcifications and dilatation of the pancreatic duct is present. Which condition is most likely considering this history and finding? A. Acute pancreatitis B. Chronic pancreatitis C. Adenocarcinoma of the pancreas D. Islet cell tumor E. Metastasis to the pancreas

B

A 75 year old man presents with ab pain, hematuria, and a palpable upper ab mass. The sonographer performs an ab scan. When scanning the kidney, a solid mass with calcifications is noted. Further investigation reveals tumor involvement in the IVC. Which of the following is the most likely diagnosis? A. Wilms' tumor B. TCC C. RCC D. Multicystic dysplastic kidney disease E. Hemangioma

C

A thin patient has been referred for US eval of the pancreas. Which of the following transducers would be best for this exam? A. 2.25 MHz phased linear array B. 3.5 MHz curved linear array C. 5.0 MHz curved linear array D. 7.5 MHz small footprint phased array 12 MHz linear array

C

What is the relationship of the splenic vein to the pancreas? A. Superior B. Anteromedial C. Posterior and caudal D. Inferior and anterior E. Ventral

C

What renal disease is associated with berry aneurysms? A. RCC B. Autosomal recessive polycystic kidney disease C. Autosomal dominant polycystic kidney disease D. Multicystic dysplastic kidney disease E. TCC

C

What structure can you use to identify the anterior aspect of the head of the pancreas? A. CBD B. CHA C. GDA D. Splenic vein E. Left renal vein

C

Which lab values would be of greatest benefit to you in evaluation of a patient with acute pancreatitis? A. Aplhafetoprotein B. Serum creatinine C. Serum amylase D. Serum albumin E. Alk Phos

C

You are performing an US on an obese patient and notice a small, hypoechoic tumor located in the tail of the pancreas. This most likely represents A. Adenocarcinoma B. Cystadenocarcinoma C. Insulinoma D. Klatskin tumor E. Pancreaticoblastoma

C

You are performing an US study to rule out the presence of a pancreatic tumor. What is the most commonly occurring malignant tumor of the pancreas? A. Cystadenocarcinoma B. Insulinoma C. Adenocarcinoma D. Gastrinoma E. Klatskin tumor

C

You have been asked to aid in staging of pancreatic cancer. Which procedure is most accurate in staging pancreatic adenocarcinoma? A. AB US B. Endoscopic US C. CT D. MRI E. plain film x-ray

C

A patient presents with severe ab pain, nausea, vomiting, ab distention, increased lipase levels and decreased hematocrit. What should the sonographer suspect? A. Hepatitis B. Duodenitis C. Cholecystitis D. Hemorrhagic pancreatitis E. Cholelithiasis

D

Infarction of the spleen is associated with all of the following except A. splenic artery thrombosis B. bacterial endocarditis C. sickle cell anemia D. a splenic cyst

D

What part of the pancreas is anterior to the SMA? A. A. Head B. Uncinate process C. Neck D. Body E. Tail

D

Where should you look for the pancreas in a patient with a pancreatic transplant? A. Pouch of Douglas B. Morrison's Pouch C. LUQ D. Iliac fossa E. Epigastrium

D

Which of the following structures is not retropertoneal? A. Aorta B. IVC C. kidneys D. Spleen E. Adrenal glands

D

Which of the following would be an indicator of pancreatic transplant rejection? A. High resistance Doppler signals B. Low resistance Doppler signals C. Heterogeneous parenchyma D. A and C E. B and C

D

You have been asked to perform an US study on a patient with a pancreatic transplant. What chronic condition does this patient probably have? A. Crohn's disease B. Chronic pancreatitis C. Lymphoma D. Type 1 diabetes mellitus E. Lupus

D

You have obtained a sagittal image of the pancreatic head and detect a small, tubular structure coursing cephalocaudad anterior to the pancreas. What is this structure? A. CBD B. SMA C. CHA D. GDA E. Left gastric artery

D

Your are reviewing a CT report on a patient referred for Ab US. The report states that pancreatic divisum is present. What does this mean? A. The pancreas is split into two pieces on each side of the abdomen B. The pancreas head is separate from the body and tail C. The pancreatic duct is duplicated D. The two pancreatic ducts have not fused E. The pancreas is malrotated

D

Five patients from a renal transplant ward are sent to US for postop sonograms. The following is a list of the indications for their postop exam. Based on these indications, which patient was not sent to a machine equipped with Doppler? A. Renal artery stenosis B. Renal vein thrombosis C. Rejection D. urinoma formation E. Acute tubular necrosis

D (Doppler findings are abnormal in renal artery stenosis, renal vein thrombosis, rejection and ATN)

Gamma Testing

Designates ideal product use test. Item is thoroughly evaluated by end user. To pass test the item must solve whatever problem customer has, no matter how long it takes.

Beta Testing

Done at customer site to determine if the product works and is bug-free

Alpha Testing

Done in-house by employees

A 39 year old renal transplant patient arrives in the US dept. The patient is referred for all of the following except A. to rule out hydrohephrosis B. to rule out fluid collections around the kidney C. Doppler imaging of the arcuate artery to rule out rejection D. Doppler imaging of the renal vein to rule out a kinked vein E. to rule out cyclosporine toxicity

E

The tail of the pancreas is in contact with which of the following structures? A. Left kidney B. Splenic flexure of the colon C. Spleen D. A and C E. All of the above

E

The typical position for scanning the spleen is A. reverse Trendelenburg B. prone C. erect D. LLD E. RLD

E

What is the name of the main duct in the pancreas? A. Duct of Santorinin B. Cystic duct C. Duct of Oddi D. Duct of Vater E. Duct of Wirsung

E

When imaging the pancreas, which vessel do you routinely visualize at the posterior border of the pancreatic head? A. AB aorta B. Right renal artery C. SMA D. SMV E. IVC

E

Which of the following is a risk factor for the development of pancreatic cancer? A. Smoking B. High-fat diet C. Diabetes D. Chronic pancreatitis E. All of the above

E

Which of the following is not an indication for renal artery Doppler imaging? A. Possible renal artery stenosis B. Hypertension C. Ab bruit D. Unilateral small kidney E. Renal calculi

E

Which structure can be seen coursing transversely at the level of the upper pancreatic head? A. CBD B. CHA C. GDA D. Splenic vein E. Left renal vein

E

You are having difficulty imaging the entire pancreas in a patient referred for AB US. Which part of the pancreas is least commonly visualized by US? A. Head B. Neck C. Uncinate process D. Body E. Tail

E

You are requested to perform an AB US on a patient to evaluate for complications of pancreatitis. What should you look for? A. Pseudoaneurysm B. Pseudocyst C. Phlegmon D. Abscess E. All of the above

E

You have been asked to rule out pseudocyst formation in a patient with acute pancreatitis. What is the US appearance of a pancreatic pseudocyst? A. Cyst without internal echoes B. Cyst with low level echoes C. Cyst with internal septations D. Well defined wall E. All of the above

E

You have documented the presence of a pseudocyst adjacent to the pancreatic head in a 56 year old male. Pseudocysts may be associated with which of the following? A. Acute pancreatitis B. Chronic pancreatitis C. Pancreatic cancer D. A and B E. All of the above

E

You have just discovered a pancreatic mass suspicious for adenocarcinoma in a patient with weight loss and ab pain. What associated findings should you look for? A. Lymphadenopathy B. Liver metastasis C. Portal vein aneurysm D. Aortic aneurysm E. A and B

E

Role of Marketing During Development

Involved from the beginning of the new product process, advises the new product team on how the product fits in with marketing capabilities and needs.

then exploit

Keep only 'Winner' for selection

Knowledge gained from product use testing

Pre-use sense reactions. Early use experiences ("Does it work?"). Major benefits results (beta tests). Diagnostic information.

Triangular Comparison Test

Product is compared to two competitive products instead of one

Monadic Test

Respondent tests single product for a period of time, services usually use this

stopping tests early

Simulations show that stopping an A/A test3 when you see a positive result will results in successful test in 41% of time

Pitfalls to Beta Testing

Site has no capacity to test the performance of product at required levels, developer uses wishy-washy performance requirements (like user-friendly) that are difficult to measure, done too late in product process, developers test own products, negative results are ignored.

negative uplift

Stopping an A/B test with even negative effect results in a win 9% of the time!

Arguments against product use testing

Suggests lack of faith, expensive, time consuming, competitors working on similar products make it to market first, consumers have to learn how to use products first

Use Testing

Testing the prototype under normal operating conditions

Sequential Monadic Test

back-to-back monadic tests with the same respondent

good power/ bad power

good = 80% bad = 30% If your power is low, then you miss many actual effects, so most of your positive results consist of false positives

B

modified in some respect (treatment)

A

the currently used version (control)

Paired Comparison Test

use of the test product is mixed with a competitive product

Pre-Use Sense Reactions

users reacting to immediate sensations of color, speed, durability, mechanical suitability, etc.