Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-05
16:37:21 jaypipes I see
16:37:39 mriedem the other thing was a latent race in the scheduler host manager stuff where it picks up newly created compute nodes from the db before they have free_disk_gb set
16:37:55 jaypipes ok
16:38:20 mriedem seems we shouldn't even pull those types of compute nodes from the database in the first place, but i'm not very familiar with that host manager code
16:39:19 jaypipes mriedem: odd that the scheduler report client would affect compute_nodes.free_disk_gb, considering the scheduler report client never looks at that field.
16:39:25 mriedem it's not that
16:39:37 jaypipes I'm reading your bug report comment.
16:39:40 mriedem update_resource_stats in the scheduler report client calls compute_node.save(),
16:39:46 jaypipes right
16:39:49 mriedem jaypipes: it's probably easier to just read the commit message
16:39:58 jaypipes :) trying to do that now...
16:39:58 mriedem as the bug report is a bit scatterbrained during debug
16:43:54 sahid sfinucan: hum :/ i commented i think we can't remove that code so i would like you to remove the TODO
16:44:26 jaypipes sahid: did you see my review comment about renaming cpu_overhead to cpus_reserved?
16:44:53 sahid jaypipes: yes, yes i did not get time to do it so far
16:45:15 jaypipes sahid: no worries, just wanted to make sure it didn't get lost in the rebases and revisions :)
16:45:39 sahid no, no don't worry - i wanted to ask you if you want me to change the name only for the field or everywhere ?
16:46:28 sahid well i guess everywhere :)
16:47:28 mriedem sdague: you may also enjoy https://review.openstack.org/#/c/416810/
16:48:29 sfinucan sahid: Ugh, yeah. Whoever thought it was a good idea to put JSON in the database :( I'll do that now
16:49:54 sahid sfinucan: yes it's a particularity of the numa part
16:50:26 jaypipes mriedem: +2 on both patches. nice work with jlvillal and JayF.
16:50:34 mriedem thanks
16:50:41 JayF mriedem did most of the work
16:50:42 jaypipes sahid: yes, everywhere :)
16:50:44 JayF I just followed along in irc
16:50:46 JayF lol
16:51:07 jlvillal jaypipes: Oh I only started the conversation. mriedem and JayF did all the heavy lifting. I had to go do a 1:1 with my manager and then leave for our holiday party :)
16:51:08 openstackgerrit Matt Riedemann proposed openstack/nova-specs: Return uuid from os-aggregates API (spec) https://review.openstack.org/415511
16:51:10 mriedem sdague: added the example response ^
16:51:25 jlvillal Big thanks to mriedem and JayF :D
16:51:47 mriedem supporting me talk to myself through debugging a problem is half the battle :)
16:52:22 mriedem johnthetubaguy: want to re-apply your +2 to https://review.openstack.org/#/c/415511/ ? i just added an api response example for GET
16:52:34 openstackgerrit Jiri Suchomel proposed openstack/nova: Retry few times when creating the initial connection to libvirtd. https://review.openstack.org/416928
16:53:23 jaypipes jlvillal: holiday party definitely more important.
16:53:28 jlvillal heh :)
16:53:30 jaypipes :)
16:54:32 JayF mriedem: quack?
16:55:56 openstackgerrit Stephen Finucane proposed openstack/nova: Fix backwards compatibility for InstanceNUMACell https://review.openstack.org/396184
17:02:34 dansmith mriedem: houston, we have a problem
17:02:44 mriedem noooooo
17:02:50 mriedem i hope it's just about that regex :)
17:02:56 openstackgerrit Derek Higgins proposed openstack/nova: libvirt: fix nova can't delete the instance with nvram https://review.openstack.org/357190
17:03:37 dansmith mriedem: so, I need to go dig, but.. there's something about cellsv1 that means we don't have hosts in the database to do the cell mappings when we're doing the rest of it
17:03:51 dansmith I remember something very faintly about it, but I don't know the details..
17:04:06 dansmith I'm thinking it's something like we won't get hosts reported to the api cell until after we've been running for a bit or something
17:04:14 dansmith which will be a major wrench in the works
17:04:43 mriedem so this is for the nova-manage discover_hosts command?
17:04:49 mriedem by hosts do you mean compute node records?
17:05:17 dansmith well, map_cell_and_hosts, but same difference yeah
17:06:04 mriedem do we get the cell mapping record?
17:06:15 dansmith no, it fails early because no hosts
17:06:30 dansmith https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1244-L1247
17:06:39 jaypipes looks like dansmith needs https://twitter.com/jaypipes/status/801422826402148353
17:07:15 mriedem jaypipes: if that's intel nfv ci it's a known broken i think
17:07:15 dansmith won't help unless I'm rechecking to see if anyone else has fixed it
17:07:34 jaypipes mriedem: intel-nfv-ci, xen ci, the list goes on :)
17:08:05 mriedem wznoinsk: i've removed intel nfv ci from voting on nova, let me know when i can reenable it
17:08:19 jaypipes dansmith, mriedem: here, this'll cheer you up: https://twitter.com/jaypipes/status/801233017993719808
17:08:23 dansmith mriedem: I'm almost wondering if we never get those records up in the api node, actually
17:08:25 dansmith johnthetubaguy: around?
17:09:51 dansmith mriedem: I guess I should see how bad it would be to continue to support the old way of listing just for cellsv1 and measure that against the pain of a real fix
17:10:48 mriedem ok i see http://logs.openstack.org/85/416585/4/experimental/gate-tempest-dsvm-cells-ubuntu-xenial/fb5df4f/logs/devstacklog.txt.gz#_2017-01-05_15_37_14_035
17:10:48 jaypipes bauzas: around? you handling the rebase of https://review.openstack.org/392569? If not, happy to do it. just lemme know.
17:11:09 dansmith mriedem: yeah
17:11:13 bauzas jaypipes: ooooh :(
17:11:26 bauzas jaypipes: I was literrally clicking on submitting the follow-up changre
17:11:39 bauzas jaypipes: hold on, rebasing then
17:11:44 jaypipes bauzas: no worries.
17:11:58 mriedem dansmith: i guess it doesn't matter if we're using nova.conf or nova-cell.conf in this context because they both point to the same dbs
17:12:17 dansmith mriedem: do they?
17:12:19 mriedem i know that devstack starts the cells stuff in a different order, i'm wondering if we're running this before the child cell has started?
17:12:27 mriedem http://logs.openstack.org/85/416585/4/experimental/gate-tempest-dsvm-cells-ubuntu-xenial/fb5df4f/logs/etc/nova/nova.conf.txt.gz
17:12:31 mriedem [database] connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova?charset=utf8
17:12:39 mriedem http://logs.openstack.org/85/416585/4/experimental/gate-tempest-dsvm-cells-ubuntu-xenial/fb5df4f/logs/etc/nova/nova-cells.conf.txt.gz
17:12:44 mriedem [database] connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell?charset=utf8
17:12:48 mriedem oh i guess not
17:12:52 mriedem so...
17:12:55 dansmith yeah, so that's worth a try
17:13:11 dansmith especially since laski's comment in there says to use nova-cell.conf
17:13:13 mriedem we should be using nova-cells.conf when running nova-manage right? to get the compute nodes from the child cell
17:13:28 mriedem comment in where?
17:13:30 dansmith mriedem: houston, we may have a problem, stand by
17:13:46 dansmith mriedem: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1305
17:13:48 mriedem dansmith: in this scenario, are you tom hanks or kevin bacon?
17:14:01 dansmith mriedem: definitely tom hanks
17:14:03 bauzas mriedem: please, gravity.
17:14:10 mriedem gravity was terrible
17:14:11 dansmith mriedem: you can be mr. bacon
17:14:19 dansmith I know you have special feelings for him
17:14:23 mriedem unless you love smug ass george clooney
17:14:27 mriedem i do love kevin bacon
17:14:40 jaypipes johnthetubaguy, mriedem: seems https://review.openstack.org/#/c/415896/ might be a devstack[-gate] failure?
17:15:07 mriedem jaypipes: yeah hit a gate failure
17:15:16 mriedem cinder v1 unit tests failed, but those were dropped
17:15:30 mriedem rechecking
17:15:31 jaypipes mriedem: something being worked on, or should I look into it?
17:15:36 jaypipes ah, k
17:15:37 mriedem should be fine
17:15:55 bauzas oh snap

Earlier   Later