Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-05
14:57:54 donnyd sean-k-mooney: do you have a log file handy?
14:58:52 sean-k-mooney donnyd: node_failure means that nodepool was not able to provision a node for the zuul job
14:59:04 sean-k-mooney so it failed before starting the job
14:59:22 sean-k-mooney there would be a log entry in nodepool but not in what is uploaded
14:59:43 donnyd http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1
14:59:57 donnyd I only see 3 failures from the last 6 hours
15:00:47 sean-k-mooney if you search for 679656 in https://zuul.opendev.org/t/openstack/status you wil see the latest one
15:01:16 sean-k-mooney this can happen if zuul is jsut at capastity for the provider
15:01:27 sean-k-mooney and it timed out waiting
15:01:47 sean-k-mooney i was seeing some node faiures before you update the lables too
15:02:08 sean-k-mooney so maybe its just worse due to the shared pool
15:02:58 sean-k-mooney it was about 1 in 2 or 1 in 3 that failed previously
15:03:03 donnyd sean-k-mooney: I am not sure where I am supposed to be looking
15:03:22 donnyd that is a pretty high failure rate no matter what
15:03:27 donnyd shouldn't be failing that often
15:04:06 donnyd maybe someone in infra can give us some insights. I just provisioned 10 without a single failure
15:04:07 sean-k-mooney well as i said it was a node failutre meaning noodpool could not provide a node to zuul to run the job on. that can happen due to hitting quota
15:04:20 efried kashyap: can we talk about https://blueprints.launchpad.net/nova/+spec/allow-secure-boot-for-qemu-kvm-guests ?
15:04:32 kashyap efried: Saddled with calls; but do ask away
15:04:41 kashyap efried: Actively working on it...
15:04:55 sean-k-mooney donnyd: FN is proably fine this could be on infras side.
15:04:57 donnyd sean-k-mooney: let me up the quota and see if that is helpful
15:04:58 efried kashyap: do you hope to land it in the next week or should I defer to ussuri?
15:05:27 sean-k-mooney i need to jump on a call but for now ill wait and check the jobs later when the load drops a bit
15:05:32 donnyd ok
15:05:54 openstackgerrit Kobi Samoray proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
15:06:16 sean-k-mooney we intend to run this job nightly at 6am with ocational manual triggering so it should not generally be a proablem
15:10:16 donnyd sean-k-mooney: infra said they were hitting quota issues, so i just oversubscribed it for instances by 30% and pulled the rest of the quota entirely
15:10:58 donnyd openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://openstack.fortnebula.com:13774/v2.1/e8fd161dc34c421a979a9e6421f823e9/servers, Quota exceeded for instances: Requested 1, but already used 70 of 70 instances
15:11:11 donnyd sean-k-mooney: that issue should be solved
15:11:37 openstackgerrit Balazs Gibizer proposed openstack/nova master: migrate: Add bw min service level check of source compute https://review.opendev.org/680395
15:11:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: resize: Add bw min service level check of source compute https://review.opendev.org/680396
15:13:25 openstackgerrit Stephen Finucane proposed openstack/nova master: Validate CPU config options against running instances https://review.opendev.org/680107
15:13:25 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Use sane indent https://review.opendev.org/680229
15:13:26 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add 'NUMACell.pcpuset' field https://review.opendev.org/680108
15:13:26 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800
15:13:27 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571
15:13:27 openstackgerrit Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801
15:13:28 openstackgerrit Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895
15:14:37 openstackgerrit Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109
15:14:53 artom dansmith, just want to make sure you saw my updates to the NUMA LM patch last night
15:15:16 dansmith artom: I did, but Imma wait until mriedem has a go through it
15:15:24 artom dansmith, ack
15:16:39 efried dustinc: I'd like to defer provider-config-file to ussuri given it has only had one reviewer (me) looking at it so far. Please let me know if you have reason to believe it can land in the next week.
15:20:30 openstackgerrit Balazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request https://review.opendev.org/655113
15:20:30 openstackgerrit Balazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware https://review.opendev.org/676138
15:20:55 kashyap efried: Don't defer it yet, please. But I'll keep you posted.
15:21:04 efried kashyap: ack
15:21:32 kashyap efried: I'll also be using that exception from Adam's InvalidMachineType patch; so there's a small dep there
15:21:56 efried kashyap: unclear whether SEV will make it too fyi
15:22:20 kashyap efried: Hmm, understandable; review bandwidth, topic complexity, etc...
15:23:10 kashyap But to state the obvious, I don't want to rush anything, though.
15:23:46 efried stephenfin: how are you feeling about cpu-resources?
15:24:42 stephenfin efried: I'm happy with it, but I need someone to take the hit on https://review.opendev.org/#/c/671800/ I've simplified it as much as I can at this point, I think
15:24:52 stephenfin I've been bugging bauzas for that :)
15:25:02 efried stephenfin: what I mean is: do you reasonably expect it can land by next Thursday?
15:25:14 stephenfin Yeah, I think so
15:25:20 efried okay
15:25:27 bauzas efried: I feel it's a reasonable target if someone else but me does reviews as well
15:25:33 bauzas hint : could be you
15:25:45 efried bauzas: I've been doing what I can, but much of it is beyond my ken.
15:25:50 bauzas cool
15:26:27 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140
15:27:02 stephenfin efried: tbh, it's probably beyond most people that aren't called me and sean-k-mooney (and maybe cfriesen)
15:27:22 stephenfin so I think we're going to have to rely on the fact that there are lot of functional tests there, and ask me all the questions if necessary
15:27:35 stephenfin (and I mean, a _lot_ of functional tests)
15:28:04 efried Okay, sean-k-mooney if you've got room, it would be helpful if you could ack that stuff.
15:28:34 efried I think between bauzas, alex_xu, sean-k-mooney, and me, we should be able to scrape together the expertise. It's more a question of bandwidth.
15:29:24 openstackgerrit Balazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth https://review.opendev.org/676972
15:32:26 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980
15:32:26 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497
15:35:15 sean-k-mooney efried: sorry was on a call
15:35:18 sean-k-mooney what was the context
15:35:53 efried sean-k-mooney: The cpu-resources series
15:36:05 bauzas sean-k-mooney: the point was whether we have reasonable amount of eyes for cpu-resources
15:36:07 efried sean-k-mooney: it would increase confidence if it had your ack
15:36:26 efried dansmith: regarding isolated vs forbidden, this is a Nova UX vs Placement API thing.
15:36:32 sean-k-mooney i can dedicate the next few days to testing it like i was toing with artoms stuff
15:36:40 dansmith efried: yeah I called that out
15:36:59 efried From the Nova perspective, we're isolating aggregates. Internally we're using placement's forbidden aggregates feature.
15:37:02 sean-k-mooney i think the numa migration is more or less in a good place so i can review the cpu serieis and start testing
15:37:28 dansmith efried: did you see my comment on the later patch?
15:38:00 sean-k-mooney efried: while i hate to do this to my own seires we could punt https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/image-metadata-prefiltering if required to help to U. its ready to go but the sev and cpu series conflict with it
15:38:20 sean-k-mooney so untill they are landed its going to keep gong into merge conflict
15:38:29 openstackgerrit Balazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize https://review.opendev.org/678827
15:39:22 efried sean-k-mooney: thanks, that's helpful
15:40:28 sean-k-mooney efried: stephen created https://etherpad.openstack.org/p/nova-cpu-resources as a set of test he wanted me to try.
15:40:54 sean-k-mooney i can start on them today/tomorrow but if anything else woudl help i could try that.
15:41:17 sean-k-mooney i can if need also try and create a temp ci job
15:41:24 efried dansmith: seeing it now. Sorry, I'm spread in a number of directions at the moment.
15:41:42 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow resizing server with port resource request https://review.opendev.org/679019
15:41:46 dansmith efried: let me circle back and explain more about why I think the seam needs to move up a step
15:41:55 efried dansmith: From my perspective, everything on the nova side should be called isolated, and we should only call placement-isms forbidden
15:42:17 dansmith efried: cool, I shall tailor my response to indicate why that's a problem
15:42:56 efried dansmith: It's a little bit like where we (should) draw the line between 'member_of' and 'aggregate'.
15:43:02 dansmith efried: yep
15:43:18 efried 'member_of' should really only start appearing in the thing we're using to construct the placement querystring.
15:43:30 efried RequestGroup, I think it's called.
15:45:32 dustinc efried: I’d really like to get it landed in train and am working towards that. I hope to get it reviewable again by eod today.
15:46:14 efried dustinc: I would like that too, but unfortunately I don't feel like it's close enough given how much other stuff we're trying to cram in in the next week.

Earlier   Later