14:30:35 #startmeeting Pulp Triage 2016-05-13 14:30:35 !start 14:30:35 Meeting started Fri May 13 14:30:35 2016 UTC and is due to finish in 60 minutes. The chair is dkliban. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:30:35 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:30:35 The meeting name has been set to 'pulp_triage_2016_05_13' 14:30:35 #info dkliban has joined triage 14:30:35 dkliban has joined triage 14:31:01 !here 14:31:01 #info ttereshc has joined triage 14:31:02 ttereshc has joined triage 14:31:10 !here 14:31:10 #info pcreech has joined triage 14:31:10 pcreech has joined triage 14:31:41 Do I need to tell it I'm here still or does it just know because I talked? 14:31:51 I think you have to register as here 14:31:51 Hmm. Guess I need to tell it. 14:31:56 #info jcline has joined triage 14:31:56 !here 14:31:57 jcline has joined triage 14:32:19 I believe we have a quorum 14:32:28 It won't let you !next without one :) 14:32:35 !next 14:32:37 #topic in Search API, restritcting fields does not work with Mongoengine - http://pulp.plan.io/issues/1332 14:32:37 16 issues left to triage: 1332, 1773, 1885, 1893, 1897, 1901, 1902, 1903, 1905, 1906, 1907, 1909, 1910, 1911, 1912, 1915 14:32:37 Pulp Issue #1332 [NEW] (amacdona@redhat.com) - Priority: High | Severity: Medium 14:32:38 in Search API, restritcting fields does not work with Mongoengine - http://pulp.plan.io/issues/1332 14:32:46 asmacdo: are you working on this ? 14:32:56 The !propose accept behavior is still confusing as hell, it's worth pretending that command doesn't exist. 14:33:03 !here 14:33:03 #info smyers has joined triage 14:33:04 smyers has joined triage 14:33:09 dklibani took a look 14:33:16 i think it needs to be a new bug 14:33:27 restricting fields does work with mongoengine 14:33:47 at least one is failing, with users, but at least repo search is working 14:33:49 asmacdo: you want to file a new issue? 14:34:01 sure, i can do that 14:34:34 asmacdo: i am going to skip this issue, but after you write a new one, you should close this one 14:34:41 ack 14:34:52 #info mhrivnak has joined triage 14:34:52 !here 14:34:52 mhrivnak has joined triage 14:34:53 +1 14:35:03 !skip 14:35:05 15 issues left to triage: 1773, 1885, 1893, 1897, 1901, 1902, 1903, 1905, 1906, 1907, 1909, 1910, 1911, 1912, 1915 14:35:05 #topic No "unprotected/http" option available for ostree repos - http://pulp.plan.io/issues/1773 14:35:06 OSTree Support Issue #1773 [NEW] (unassigned) - Priority: Normal | Severity: Medium 14:35:06 No "unprotected/http" option available for ostree repos - http://pulp.plan.io/issues/1773 14:35:24 we have skipped this one for about a month now 14:35:47 yep 14:35:49 ya. and jortel is tied up right now, so he can't comment. 14:35:58 skip it again? 14:36:05 what do we need to triage it? 14:36:20 we need scope. 14:36:30 but honestly, this should probably be a story. 14:36:43 I can see that 14:36:52 I don't think it's a bug. It's lack of a behavior that they might want. 14:36:53 does it need scope from jortel or someone else? 14:36:56 It's not clear to me that this issue is blocked, since partha has replied since common #2 14:37:01 comment 14:37:15 need scope from a mix of jortel and partha probably. 14:37:28 I'm in favor of making it a story, and leaving it Normal/Medium for now 14:37:33 +1 14:37:36 +1 14:37:41 If it needs grease, it will squeak. :) 14:37:41 !here 14:37:41 #info bowlofeggs has joined triage 14:37:42 bowlofeggs has joined triage 14:37:42 ok ... i am making it a story 14:37:56 Fine with me 14:38:02 !next 14:38:04 #topic Pulp uploads are failing with "too many open files" error - http://pulp.plan.io/issues/1885 14:38:04 14 issues left to triage: 1885, 1893, 1897, 1901, 1902, 1903, 1905, 1906, 1907, 1909, 1910, 1911, 1912, 1915 14:38:05 Pulp Issue #1885 [NEW] (unassigned) - Priority: Normal | Severity: Medium 14:38:06 Pulp uploads are failing with "too many open files" error - http://pulp.plan.io/issues/1885 14:38:37 mhrivnak: this sounds bad 14:38:57 But then in the comments, not so bad 14:38:58 It does, but I couldn't reproduce it, and it sounds like they are having trouble reproducing it. 14:39:10 Not sure if pulp bug or RCM-patched-pulp bug 14:39:28 we could wait a week 14:39:35 this is not a priority for them - I had a conversation with dgregor regarding it 14:39:36 mhrivnak, where did the number 6000 come from? 14:39:44 I made it up. 14:39:46 Cool 14:39:52 I don't think this is an actual bug 14:39:59 should we close it or skip it? 14:40:00 I agree 14:40:10 close not a bug? 14:40:22 I think so and I can do that with a comment on it 14:40:23 I want to close it notabug pending a reproducer 14:40:29 So I don't know much about upload, but don't we do it in chunks (not the HTTP chunked feature)? 14:40:29 bmbouter: thanks! 14:40:42 Could that lead to a bunch of partial uploads with open file handles? 14:40:55 it could, but after testing we couldn't reproduce 14:41:06 The chunks should all be in-memory, streamed to a single file handle, in my experience. 14:41:09 If the files uploaded are smaller than the chunk size maybe it wouldn't happen 14:41:20 smyers, the chunks come in different requests. 14:41:22 Okay, I'm out of ideas then 14:41:41 let's close but invite reproduction steps 14:41:45 In any case, I agree that closing and inviting a reproducer would be good. 14:41:46 I want to fix it if its broken, but I don't see how without a reproducer 14:41:46 maybe close as works for me? 14:42:10 +1 14:42:10 Yeah, I think worksforme is a more appropriate status 14:42:11 alright .. bmbouter is leaving a comment and closing it. we are moving to the next bug. 14:42:12 good call 14:42:13 +1 14:42:15 +1 14:42:17 doing it now 14:42:21 !next 14:42:23 13 issues left to triage: 1893, 1897, 1901, 1902, 1903, 1905, 1906, 1907, 1909, 1910, 1911, 1912, 1915 14:42:24 #topic pulp_docker does not support Docker < 1.10 with manifests that were published from Docker >= 1.10 - http://pulp.plan.io/issues/1893 14:42:24 Docker Support Issue #1893 [NEW] (unassigned) - Priority: Normal | Severity: Medium 14:42:24 pulp_docker does not support Docker < 1.10 with manifests that were published from Docker >= 1.10 - http://pulp.plan.io/issues/1893 14:42:39 * smyers facepalms 14:42:50 this one is still unknown for priority 14:42:55 unless jalbertson knows more 14:43:14 bowlofeggs, I do not 14:43:23 last i heard was that RCM was going to test a matrix of versions to see if the problem is immediate or not 14:43:33 i had a meeting with them yesterday 14:43:39 and they said they had not tested this yet 14:43:55 we definitely have to do something, but the question is really just over priority 14:44:08 It seems triagable based on that, since it's apparently not being treated as High priority 14:44:30 i think we should make it Normal priority 14:44:31 true 14:44:34 +1 14:44:41 any objections? 14:44:45 +1 14:44:48 Severity might be High, since pulp just doesn't work in this case 14:45:07 High severity, Normal priority 14:45:21 I'm still a little uncertain of the impact on users. 14:45:29 * smyers too 14:45:33 Having that matrix of test results would be very helpful. 14:46:02 So normal/med prio and sev 14:46:05 ? 14:46:22 I'm not sure we can know the priority without better understanding of the impact. 14:46:36 !propose needinfo 14:46:36 #idea Proposed for #1893: This issue cannot be triaged without more info. 14:46:37 Proposed for #1893: This issue cannot be triaged without more info. 14:46:51 mhrivnak: do you proopose that we skip it and leave it untriaged? 14:47:09 someone needs to take action on it though 14:47:09 I think that would be a fine option. 14:47:23 Sounds like RCM is going to do some testing? 14:47:26 rcm had volunteered to do that testing 14:47:30 ok 14:47:34 let's skip then 14:47:40 any objection to skipping? 14:47:57 I'll add a note asking for more info. 14:48:03 mhrivnak: thank you ! 14:48:10 !skip 14:48:12 #topic catalog entries not created for pre-existing units - http://pulp.plan.io/issues/1897 14:48:12 11 issues left to triage: 1897, 1901, 1902, 1903, 1905, 1906, 1907, 1910, 1911, 1912, 1915 14:48:12 RPM Support Issue #1897 [ASSIGNED] (jortel@redhat.com) - Priority: Normal | Severity: Medium 14:48:13 catalog entries not created for pre-existing units - http://pulp.plan.io/issues/1897 14:48:28 i think this was already triaged 14:48:29 ah, we can skip that, jortel accidentally untriaged it 14:48:32 I kinda like what's going on here, where the guy running triage isn't always the guy changing the issue 14:48:57 So just add the triage check back? 14:49:04 dkliban, I'll mark it as triaged again 14:49:10 jcline: thank you ! 14:49:17 yeah, sorry .. didn't mean to untriage. no idea how it happened 14:49:22 !next 14:49:24 #topic Fix error handling during the erratum update - http://pulp.plan.io/issues/1901 14:49:24 10 issues left to triage: 1901, 1902, 1903, 1905, 1906, 1907, 1910, 1911, 1912, 1915 14:49:24 RPM Support Issue #1901 [NEW] (unassigned) - Priority: Normal | Severity: Medium 14:49:25 Fix error handling during the erratum update - http://pulp.plan.io/issues/1901 14:50:02 ttereshc: are you working on this already? 14:50:12 no 14:50:29 is Normal priority, Medium severity appropriate for this? 14:50:48 I think so 14:50:50 sounds important to me 14:51:10 Only if there's malformed errata running around 14:51:16 If we run into errata in the wild that can't be parsed for this reason, it'll be important to get clear feedback. 14:51:22 oh wait 14:51:24 Which shouldn't happen, and if it does you see this rather than a nicer message 14:51:30 it's the malformed data one from last week 14:51:34 jcline++ 14:51:34 jcline's karma is now 1 14:51:35 I think the error message is pretty clear 14:51:49 #idea Proposed for #1901: Leave the issue as-is, accepting its current state. 14:51:49 !propose accept 14:51:49 Proposed for #1901: Leave the issue as-is, accepting its current state. 14:51:54 I'm thinking that to change the error handling here might basically mean changing how pulp-admin displays errors 14:51:58 dkliban: that was a real propose accept! 14:52:04 You just have to go digging for the error message, rather than seeing it right away. 14:52:05 Which is a pretty huge deal 14:52:24 smyers, I don't think that's likely the case. 14:52:30 Yeah, but that's true of tons of pulp-admin errors 14:52:48 I'm not saying it's good, obviously. Just common. 14:52:54 well, maybe it's the case on this command specifically? 14:52:56 Yeah, we can fix just this one, but it'll be just another one-off fix 14:53:04 (jaoof) 14:53:10 Usually we can just raise a PulpCodedException, and the right thing happens. 14:53:21 no client-side work required. 14:53:37 Is PulpExecutionException not a PulpCodedException? 14:53:44 But it's possible that isn't supported everywhere in pulp-admin yet? I don't recall the scope of that. 14:53:48 (That would be appropriately hilarious) 14:53:58 I think it s also good not to fail on the first erratum but try to update all of them 14:54:03 smyers, I would not be surprised if it isn't 14:54:10 * smyers too 14:54:19 Anyway, I'm fine with normal medium or normal low 14:54:21 don't think it is. 14:55:03 Normal/Medium .... any objections? 14:55:18 do it 14:55:18 works for me. 14:55:29 +1 14:55:35 !next 14:55:37 9 issues left to triage: 1902, 1903, 1905, 1906, 1907, 1910, 1911, 1912, 1915 14:55:37 #topic upload performance degrades as repo size increases - http://pulp.plan.io/issues/1902 14:55:38 RPM Support Issue #1902 [NEW] (unassigned) - Priority: Normal | Severity: Medium 14:55:38 upload performance degrades as repo size increases - http://pulp.plan.io/issues/1902 14:55:40 Probably a pretty easy fix, so someone could tack it on at the end of a sprint if they like, or a Friday afternoon. :) 14:55:51 It's also probably worth a comment from mhrivnak including the PulpCodedException info on #1901 14:56:01 I'll add that comment. 14:56:17 I feel #1902 is important for large pulp installs that do lots of uploading 14:56:25 1902 is a regression in 2.8. It makes upload real bad. 14:56:26 yeah 14:56:40 suggest High prio 14:56:42 i say this is High / High ? 14:56:43 !propose high med 14:56:43 Error: "propose" is not a valid command. 14:56:47 * bmbouter is not sure how to propose 14:56:53 #idea Proposed for #1902: Priority: High, Severity: Medium 14:56:53 !propose triage high med 14:56:53 Proposed for #1902: Priority: High, Severity: Medium 14:56:57 let's not use that feature. etherpad says so. 14:56:58 This is pulp-admin rpm upload behavior? 14:56:59 Either high high or high medium 14:57:11 etherpad? 14:57:23 I only tested it with pulp-admin, but the regression is on the server side. 14:57:37 Because what's slowing this down is likely duplicate nevra protection, which on upload happens per-unit, and will progressively slow down as more units are added. 14:57:42 the API performance degrades 14:57:48 oh. 14:57:51 i don't think it's high severity - it doesn't crash, just gets slow 14:57:57 Yeah. If I'm right, then we want this. 14:58:04 It's slower for a very good reason. 14:58:23 smyers: makes sense ... i just thoguht our indexes would help alliviate the problem 14:58:28 I hadn't thought of that 14:58:32 me neither 14:58:36 The task went from averaging maybe 50-100ms to averaging almost 5 seconds as the repo got large. 14:58:43 They do, I think. It'd be way worse without them. 14:58:48 still though it's important so I suggest we give it high prio 14:58:50 That's going to be rough on people with hundreds of thousands of rpms though, right? 14:58:53 * bmbouter has to go, ttyl 14:59:07 Are people uploading hundreds of thousands of rpms one at a time? 14:59:16 Maybe createrepo that pile and sync it the normal way? 14:59:18 smyers: yeah ... i think tht's how it works 14:59:22 can't you only upload rpms one at a time? 14:59:30 it's one at a time. 14:59:38 Well maybe hundreds of thousands, but I wouldn't be surprised if it was thousands at a time 14:59:54 Perhaps we can do the nevra protection a faster way for upload than for sync. 15:00:07 thousands at a time definitely happens. 15:00:36 bowlofeggs, by "at a time" I just meant in a batch 15:01:15 I think this is worth a little investigation in the near term. 15:01:29 so high priority / medium severity? 15:01:35 +1 15:01:45 the postgres work will solve this problem 15:01:57 because proper db design is really what's lacking here 15:01:57 yeah ... but we are not even close to that 15:02:10 so let's see what we can do in the short term 15:02:17 high/med works for me. 15:02:26 great! i am marking it as such 15:02:40 !next 15:02:42 8 issues left to triage: 1903, 1905, 1906, 1907, 1910, 1911, 1912, 1915 15:02:42 #topic RPM import traceback (non-utf-8 metadata slipping through) - http://pulp.plan.io/issues/1903 15:02:43 Pulp Issue #1903 [NEW] (unassigned) - Priority: Normal | Severity: Medium 15:02:43 RPM import traceback (non-utf-8 metadata slipping through) - http://pulp.plan.io/issues/1903 15:03:36 ah yes. We have some protection during sync against rpms with non-utf8 encoding, but no such protection during upload. 15:03:39 i blieve RPMs are required to be UTF-8 15:03:51 RPMs do not have a field to indicate what their encoding is 15:03:56 looks like there is a patch already 15:03:59 so they need to be UTF-8 15:04:21 latin-1 is not valid in an RPM afaik 15:04:30 i had a tech-list thread about this a few years ago 15:04:35 the reporter was doing some work to try to fix it, so I added that patch as a quick way to help him/her move forward. 15:05:06 that patch is not safe 15:05:18 if you receive invalid encoding, it's not a good idea to assume that it's latin-1 15:05:25 I think the rpms in question are from suse, and suse themselves have a policy that all rpms must be utf-8 15:05:31 but the rpms are old enough that they don't want to fix them. 15:05:31 that can and will result in corrupted data in the database 15:05:52 there are hundreds of codecs 15:05:57 Indeed. We did have a lot of discussion about this a long time ago when the protection was added to the sync workflow. 15:06:08 latin-1 is a popular one, but there ar emany others in use, especially in asia (the most populous continent) 15:06:27 this patch would corrupt asian data 15:06:42 it's better for us to fail with a good error message 15:06:46 FWIW, this patch would have upload do exactly what sync already does, so maybe we need to re-consider that behavior too? 15:06:47 about the rpm having invalid data 15:07:09 regardless of the patch, do we want to triage the problem and then have someone review and determine whether or not the patch is the right solution 15:07:25 or do we not think this is an issue 15:07:53 I think that's the problem. It appears to boil down to "Do we want to fix pulp to allow uploading invalid RPMs?" 15:07:55 i think we should mkae the issue be about bad error messages 15:08:16 If we do that, we need to also consider changing how sync handles the same situation. 15:08:46 or should we skip and have people add comments to the issue and re-evaluate? 15:08:57 Sounds like we need to nail down the "right behavior" first, and then make sure both places do it. 15:09:24 so i propose that bowlofeggs and mhrivnak leave some comments on the issue 15:09:29 we will skip it for today 15:09:40 That's a new task, right? For deciding the right option? 15:10:00 let's start by having a conversation in the comments of this issue though 15:10:06 +1 15:10:38 mhrivnak and bowlofeggs ... does that work for you? and we'll move on to the next issue 15:10:42 works for me. 15:10:56 +1 15:10:59 !next 15:11:02 7 issues left to triage: 1905, 1906, 1907, 1910, 1911, 1912, 1915 15:11:03 #topic Streamer SSL cert not added to Pulp's trust store on RHEL 6 - http://pulp.plan.io/issues/1905 15:11:03 Packaging Issue #1905 [NEW] (unassigned) - Priority: Normal | Severity: Medium 15:11:03 Streamer SSL cert not added to Pulp's trust store on RHEL 6 - http://pulp.plan.io/issues/1905 15:11:30 ipanova: ++ 15:11:30 ipanova: 's karma is now 1 15:11:57 To summarize this, the CA used for apache doesn't end up the trust store on rhel6, but it does everywhere else. 15:12:13 sounds like a very bad problem 15:12:20 It's an easy fix, there must just be some difference in RHEL 6. 15:12:26 this is for CI only, fwiw 15:12:28 This is just for our test infrastructure 15:12:31 i say high/med 15:12:40 high/low, even? 15:12:45 yeah high/low 15:12:47 either is fine with me 15:12:57 high prio, low sev, to be clear 15:13:01 !propose high low 15:13:01 Error: "propose" is not a valid command. 15:13:05 :( 15:13:09 !propose triage high low 15:13:09 #idea Proposed for #1905: Priority: High, Severity: Low 15:13:10 Proposed for #1905: Priority: High, Severity: Low 15:13:19 https://en.wikipedia.org/wiki/High/Low_%28Nada_Surf_album%29 15:13:19 +1 15:13:20 Title: High/Low (Nada Surf album) - Wikipedia, the free encyclopedia (at en.wikipedia.org) 15:13:41 works for me 15:13:54 i am marking it as High priority Low severity 15:14:22 !next 15:14:24 6 issues left to triage: 1906, 1907, 1910, 1911, 1912, 1915 15:14:24 #topic Consumers are added to group but not bound to group repos - http://pulp.plan.io/issues/1906 15:14:25 RPM Support Issue #1906 [NEW] (unassigned) - Priority: Normal | Severity: Medium 15:14:25 Consumers are added to group but not bound to group repos - http://pulp.plan.io/issues/1906 15:14:44 our consumer and repo grouping features have always been pretty half-done 15:14:50 is this a story or an isuse? 15:14:52 1/4 done, really 15:14:53 this is kind of a story 15:15:04 agreed. 15:15:10 because it's like "why did you make this feature when you can't do hardly anything with it?" 15:15:13 +1 story 15:15:29 yeah switch to story, but we also should really actually do it 15:16:11 bowlofeggs: do you propose it for a sprint candidate? 15:16:15 We may not have a need for consumer groups in pulp 3, FWIW. 15:16:47 i think marking it as a story is enough for right now. if anyone has particular interest, bring it up at sprint planning 15:16:53 Or if we do, a re-think is certainly appropriate. 15:17:24 dkliban, agreed 15:17:36 i think it is dup 15:17:52 let's move on 15:18:04 in general, i think we can do triage faster than this 15:18:07 ipanova: if you think so, please link it to the other issue 15:18:13 !next 15:18:13 dkliban: working on this 15:18:14 5 issues left to triage: 1907, 1910, 1911, 1912, 1915 15:18:15 #topic python-pulp-streamer is not pulled with groupinstall pulp-server-qpid - http://pulp.plan.io/issues/1907 15:18:15 Pulp Issue #1907 [NEW] (unassigned) - Priority: Normal | Severity: Medium 15:18:15 python-pulp-streamer is not pulled with groupinstall pulp-server-qpid - http://pulp.plan.io/issues/1907 15:18:22 ipanova: thanks! 15:18:26 #idea Proposed for #1907: Leave the issue as-is, accepting its current state. 15:18:26 !propose accept 15:18:26 Proposed for #1907: Leave the issue as-is, accepting its current state. 15:18:37 actually 15:18:38 This is a docs bug, not normal normal is fine 15:18:39 let's not do this 15:18:59 shoudln't the streamer not always get installed? 15:19:05 it's a separate thing 15:19:05 That's what I thought 15:19:10 same here 15:19:11 me too 15:19:12 There's a whole bunch of conversation about that, yes 15:19:13 NOTABUG 15:19:28 Ultimately we decided it should be better documented 15:19:39 yeah ... i think it should be a documentation bug 15:19:49 sounds good. 15:20:06 notabug +21 15:20:09 Probably an important one, though, since the confusion causes breakage 15:20:22 let's move on 15:20:27 10 min left 15:20:39 dkliban: #1706 should be closed as dup of https://pulp.plan.io/issues/167 15:20:41 Title: Story #167: [RFE] new consumers do not auto subscribe to already bound consumer group repos - Pulp (at pulp.plan.io) 15:20:42 hold on - are you marking it as a documentation bug and accepting it or rejecting? 15:20:46 I'm thinking don't close it, just add docs and easy fix, high prio, normal sev 15:21:05 +1 to smyers' suggestion 15:21:05 i am accepting it as a documentation bug with Normal priority Medium severity 15:21:11 Docs bug and accepting it is what should happen, I think 15:21:14 +1 15:21:15 high prio 15:21:22 We broke a dude with bad docs 15:21:29 !propose triage high med 15:21:29 #idea Proposed for #1907: Priority: High, Severity: Medium 15:21:30 Proposed for #1907: Priority: High, Severity: Medium 15:21:36 !accept 15:21:36 #agreed Priority: High, Severity: Medium 15:21:37 Current proposal accepted: Priority: High, Severity: Medium 15:21:38 4 issues left to triage: 1910, 1911, 1912, 1915 15:21:39 #topic Errata update fails when id of the repo is added to the existing collection - http://pulp.plan.io/issues/1910 15:21:39 RPM Support Issue #1910 [POST] (ttereshc) - Priority: Normal | Severity: Medium | Target Release: 2.8.4 15:21:39 Errata update fails when id of the repo is added to the existing collection - http://pulp.plan.io/issues/1910 15:21:59 POST 15:22:01 !propose accept 15:22:01 #idea Proposed for #1910: Leave the issue as-is, accepting its current state. 15:22:02 Proposed for #1910: Leave the issue as-is, accepting its current state. 15:22:10 yeah let's !next 15:22:30 +1 15:22:31