| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 21:03:46 | tonyb | mikal, sdague, melwitt: I don't knwo if this email was wider distrubuted but you know how we moved last_bytes recently .. it seems it was used by nova-lxd | |
| 21:04:15 | melwitt | I think I saw that email | |
| 21:04:17 | sdague | tonyb: they are out of tree, kind of don't care | |
| 21:04:31 | tonyb | mikal, sdague, melwitt: having the (out of tree) nova-lxd driver call into the libvirt code isn't cool :( so shoudl I revert it? | |
| 21:04:42 | tonyb | sdague: Well that was my initial response | |
| 21:04:53 | edmondsw | mriedem I started a change to get the powervm driver up faster but put it aside when we closed things down for pike | |
| 21:05:04 | tonyb | sdague: especially as when we get to queens we move it again | |
| 21:05:09 | sdague | tonyb: no, show up and interact in the community if you are using internal functions in the rest of the tree | |
| 21:05:10 | melwitt | tonyb: didn't they say they're already copy-pasting it somewhere? | |
| 21:05:27 | mriedem | edmondsw: i don't see how nova has control over how fast the backend node comes up | |
| 21:05:32 | tonyb | melwitt: not in the email I have but there may be more | |
| 21:05:40 | mriedem | edmondsw: you were just working on auto-enable the service i thought | |
| 21:05:52 | melwitt | okay, lemme check. maybe I misunderstood it | |
| 21:05:55 | edmondsw | mriedem https://review.openstack.org/#/c/471773/ | |
| 21:07:00 | edmondsw | should complete init_host much faster | |
| 21:07:25 | sdague | tonyb: I wasn't on any such email, but my patience is low for out of tree driver that's not working in the community | |
| 21:08:12 | melwitt | tonyb: okay, I just saw it was the last comment in the review https://review.openstack.org/#/c/472228/ and he's saying that they've now copy-pasted it, not that they had been until now. so I misread it | |
| 21:08:20 | tonyb | okay so we more or less want to say, "Sorry. this is part of the move to privsep so until xxx merged you'll just need to work around it in your driver' | |
| 21:09:01 | tonyb | melwitt: Thanks. | |
| 21:09:15 | tonyb | I'll contact them after breakfast. | |
| 21:09:30 | melwitt | yeah, if it's going to be available again in privsep after the work is done, that seems not so bad to me | |
| 21:09:31 | tonyb | I just wanted to make check we're on the same page | |
| 21:09:37 | tonyb | cool | |
| 21:13:35 | sdague | tonyb: yes, also, work upstream :P | |
| 21:23:25 | efried | sdague mriedem Okay, we've figured out how that change affected us. We need to do some stuff to the systemctl service file and restart compute. As of now, we're doing that after devstack finishes, not caring that compute wasn't coming up all the way during the stack. With this change, it started to matter that compute wasn't starting. | |
| 21:23:52 | efried | So we're going to figure out how to tweak the service file before stacking starts. Should make compute come right up during the stack process. | |
| 21:24:28 | mriedem | live migration job is done on my sanity check change http://logs.openstack.org/87/488187/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/482b6f4/ | |
| 21:27:20 | mriedem | oh i've got a bug in there | |
| 21:33:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Sanity check delete_allocation_for_instance https://review.openstack.org/488187 | |
| 21:35:45 | dansmith | jaypipes: https://bugs.launchpad.net/nova/+bug/1707071 | |
| 21:35:46 | openstack | Launchpad bug 1707071 in OpenStack Compute (nova) "Compute nodes will fight over allocations during migration" [Undecided,New] | |
| 21:35:52 | dansmith | dude how awesome is that bug number? | |
| 21:36:03 | dansmith | palindromic and all primes | |
| 21:37:49 | dansmith | well, I guess zero isn't a prime.. damn | |
| 21:40:37 | mriedem | hmm, so on rebuild to the same host | |
| 21:40:54 | mriedem | conductor calls select_destinations | |
| 21:41:11 | mriedem | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L770 | |
| 21:41:23 | mriedem | will we double allocate that host then? | |
| 21:43:27 | mriedem | oh nvm | |
| 21:43:29 | mriedem | that's if not host | |
| 21:44:51 | mriedem | whew | |
| 21:47:44 | smcginnis | dansmith: You'll have to try to catch 1737371 | |
| 21:52:15 | dansmith | smcginnis: yeah | |
| 21:56:34 | melwitt | time to buy a lotto ticket | |
| 22:07:58 | mriedem | jaypipes: dansmith: ok, https://review.openstack.org/#/c/483566/ | |
| 22:08:06 | mriedem | there is one thing in there that worries me | |
| 22:08:19 | mriedem | https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@171 | |
| 22:09:31 | dansmith | hrm | |
| 22:10:13 | mriedem | oh L86 | |
| 22:10:13 | mriedem | if len(selected_hosts) < num_instances: | |
| 22:10:42 | dansmith | what happens if we return less than enough hosts to whatever calls us? | |
| 22:11:00 | dansmith | heh yeah that | |
| 22:11:30 | dansmith | and min/max_instances is only a quota check, so no problem there | |
| 22:12:29 | mriedem | ok +2 | |
| 22:12:35 | mriedem | i tried my hardest | |
| 22:12:39 | mriedem | to find fault | |
| 22:14:08 | dansmith | ack | |
| 22:14:31 | melwitt | mriedem: docstring doesn't match params here https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@252 if you wanted something :P | |
| 22:14:37 | mriedem | there might still be something to https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@208 | |
| 22:14:59 | mriedem | where if we know allocation requests are constantly failing for a host, we should stop trying it | |
| 22:15:17 | mriedem | omg | |
| 22:15:19 | mriedem | -10 | |
| 22:15:36 | melwitt | hehe | |
| 22:15:38 | mriedem | you know i think i was looking for where cn_uuid was used in there too | |
| 22:15:39 | mriedem | b/c of the docstring | |
| 22:18:53 | mriedem | pike-3 tag https://review.openstack.org/#/c/488218/ | |
| 22:21:09 | mriedem | jaypipes: thanks for hanging in there | |
| 22:21:24 | mriedem | ooo just in time as laura rolls back up to the house with the kid | |
| 22:21:37 | mriedem | time to get my county fair food eating clothes on | |
| 22:21:43 | dansmith | heh | |
| 22:21:52 | mriedem | over sized drawers, bib, etc | |
| 22:24:41 | mriedem | sdague: the compute host wait thing also blew up the cellsv1 job http://logs.openstack.org/87/488187/2/check/gate-tempest-dsvm-cells-ubuntu-xenial/8ac25f1/logs/devstacklog.txt.gz | |
| 22:28:03 | mriedem | dansmith: jaypipes: here is the sanity check in action on the live migration job http://logs.openstack.org/87/488187/2/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/107a810/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_27_22_21_14_766229 | |
| 22:28:26 | mriedem | i didn't see any cases where the source node deleting the allocation isn't in the list of current allocations | |
| 22:28:36 | mriedem | but it's definitely stomping over the 'double up' allocatoins | |
| 22:29:04 | mriedem | Removing allocations for instance which are currently against more than one compute node resource provider. Current allocations: {u'13b1e5e0-66ef-4533-9a07-b1a3220d6b00': {u'generation': 8, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}, u'7aa9619d-db83-4da9-b822-f4d66e7143f8': {u'generation': 6, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}} | |
| 22:33:20 | dansmith | okay I think that's just getting lucky, | |
| 22:33:29 | dansmith | as I think it could either win or lose that race, but cool | |
| 22:37:45 | mriedem | oh no you didn't | |
| 22:37:45 | mriedem | https://www.youtube.com/watch?v=mBluR6cLxJ8 | |
| 22:44:32 | melwitt | dunno | |
| 22:44:54 | mikal | I'm going to be a bad man and reply to him cc'ing openstack-dev I suppose | |
| 22:45:06 | mikal | I don't want to exclude the nova team from the conversation | |
| 23:35:57 | tonyb | mikal: Yeah the only email was the private one but you've seen the conversation here and on the review | |
| 23:36:54 | tonyb | mikal: I don't see you reply did you send it already? | |
| 23:43:02 | mikal | tonyb: I cc'ed you and sent it to -dev | |
| 23:43:26 | mikal | Subject: Re: [Nova] Broken nova-lxd | |
| 23:44:00 | mikal | http://lists.openstack.org/pipermail/openstack-dev/2017-July/120259.html | |
| 23:44:01 | tonyb | mikal: okay I don't see it yet probably stuck in a mail queue somewhere | |
| 23:50:47 | openstackgerrit | Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/488125 | |
| #openstack-nova - 2017-07-28 | |||
| 00:26:52 | openstackgerrit | Dan Smith proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | |
| 00:26:53 | openstackgerrit | Dan Smith proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | |
| 00:36:26 | openstackgerrit | Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229 | |
| 00:36:26 | openstackgerrit | Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166 | |
| 00:36:27 | openstackgerrit | Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972 | |
| 00:36:27 | openstackgerrit | Michael Still proposed openstack/nova master: rootwrap.d cleanup mislabeled files https://review.openstack.org/486831 | |
| 00:55:46 | jgriffith | mriedem_afk smcginnis ildikov I guess I'll have to accept defeat :( | |
| 00:56:34 | jgriffith | mriedem_afk smcginnis ildikov I updated the comments in the review but there's something bigger going on it looks like, inparticular instances going to error state and libvirt failing to clean them up | |
| 00:56:36 | jgriffith | http://logs.openstack.org/85/330285/106/check/gate-tempest-dsvm-py35-ubuntu-xenial/40072bb/logs/screen-n-cpu.txt.gz?level=TRACE#_Jul_26_09_34_33_062305 | |
| 00:56:48 | openstackgerrit | Dan Smith proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | |