Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-25
20:37:51 mriedem i'm not sure no
20:37:52 mriedem i can dig
20:38:06 mriedem my shovel is going to be pretty gd blunt after the end of this day
20:38:15 sdague mriedem: I was assuming that virt driver init host just took that long to start
20:38:28 mriedem the virt driver init_host doesn't do much
20:38:44 mriedem at least for the libvirt driver, it registers event listeners and connects to libvirt
20:39:11 mriedem ok so this is a subnode starting up
20:39:12 mriedem http://logs.openstack.org/79/487179/1/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/1dd9db5/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_25_19_30_42_125360
20:39:16 mriedem Jul 25 19:30:42.125360 ubuntu-xenial-2-node-osic-cloud1-disk-10073644-745537 nova-compute[711]: INFO nova.service [-] Starting compute node (version 16.0.0)
20:39:30 mriedem that's in nova.service.Service.start()
20:40:15 mriedem then you see the libvirt event stuff
20:41:03 mriedem then you see this from the libvirt driver http://logs.openstack.org/79/487179/1/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/1dd9db5/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_25_19_30_42_150823
20:41:10 mriedem because the compute node doesn't exist yet
20:41:41 mriedem then you see this in the compute manager http://logs.openstack.org/79/487179/1/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/1dd9db5/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_25_19_30_44_394800
20:41:49 mriedem which is from _get_compute_nodes_in_db
20:43:18 mriedem i'm not sure why that traces
20:43:58 mriedem that warning is here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6627
20:47:28 mriedem anyway then we call the resource tracker https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6602
20:47:44 mriedem http://logs.openstack.org/79/487179/1/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/1dd9db5/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_25_19_30_44_399281
20:47:52 mriedem Jul 25 19:30:44.399281 ubuntu-xenial-2-node-osic-cloud1-disk-10073644-745537 nova-compute[711]: DEBUG nova.compute.resource_tracker [None req-9286123e-31d0-45c4-a951-fe1e3947db00 None None] Auditing locally available compute resources for ubuntu-xenial-2-node-osic-cloud1-disk-10073644-745537 (node: ubuntu-xenial-2-node-osic-cloud1-disk-10073644-745537) {{(pid=711) update_available_resource /opt/stack/new/nova/nova/compute/res
20:47:52 mriedem e_tracker.py:609}}
20:49:41 mriedem then we should get in here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L488
20:50:43 mriedem and we'll hit this https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L709
20:50:57 mriedem http://logs.openstack.org/79/487179/1/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/1dd9db5/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_25_19_30_44_441734
20:51:39 mriedem and so we'll create the compute node record here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L531
20:51:58 mriedem http://logs.openstack.org/79/487179/1/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/1dd9db5/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_25_19_30_44_459686
20:52:27 mriedem sdague: but does systemd wait or does it just launch off the service start and not block on it?
20:52:50 sdague mriedem: define wait
20:55:20 sdague mriedem: we're starting in the foreground, it's tracking the parent process, but the issue is it's much later that things are ready
20:56:21 sdague anyway, I need to work on dinner, I've got this half assed patch running locally, if it works I'll push it
20:56:24 mriedem right when we start the n-cpu service, and the service is launched, that's all async
20:56:31 sdague mriedem: it's not async
20:56:38 jangutter mriedem, jaypipes: should I assert on exception.NovaException or exception.InternalError at https://review.openstack.org/#/c/486426/6/nova/tests/unit/virt/libvirt/test_vif.py@1616
20:56:49 sdague it's that it's not ready for 25 - 30 seconds after start
20:57:00 sdague and, there is no /health to know that
20:57:19 sdague we poll api processes that we start to know they are ready before we move on
20:57:26 sdague but there isn't a direct interface for that
20:57:34 mriedem right i meant https://github.com/openstack/nova/blob/master/nova/service.py#L138
20:57:47 mriedem which is what calls compute manager pre_start_hook that sets this all up
20:58:09 sdague mriedem: ok, before I leave, I want to make sure we get this question clear :)
20:58:30 sdague systemd is starting things, and it's running as parent process as soon as python exec happens
20:59:00 mriedem i guess i was thinking about like sysv init scripts and services,
20:59:05 mriedem where you can run service nova-compute status
20:59:08 mriedem and see if it's started or not
20:59:19 sdague sure, but all that tells you is if the process is running
20:59:25 sdague the process is running
20:59:38 sdague eventually the process is ready
20:59:42 mriedem but can't the start routine block until it's actually started or crashed?
20:59:51 sdague it is started
20:59:58 sdague the process is running
21:00:27 sdague how does anything external know if a process is ready other than pid existing?
21:00:31 mriedem ok, well this is all latent stuff and shouldn't block https://review.openstack.org/#/c/477556/
21:00:35 mriedem so can we get that in?
21:00:54 mriedem like, this behavior goes back to ocata
21:01:04 sdague https://www.freedesktop.org/software/systemd/man/systemd-notify.html if you want deeper state interaction between process and systemd
21:01:34 sdague mriedem: the neutron folks are currently borked?
21:01:39 mriedem no
21:01:49 mriedem the dvr-ha multinode job is non-voting and in the experimental queue
21:01:54 sdague ok
21:01:55 mriedem i've already talked to haleyb about it
21:02:07 sdague if they are cool with it, that's fine
21:02:44 sdague I'll try to get this wait call in place
21:02:52 sdague I just appoved the fleet patch
21:03:01 mriedem ok
21:03:13 sdague this other thing takes a while to run, so off for the night, we'll see what it looks like in the morning
21:06:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove redundant free_vcpus logging in _report_hypervisor_resource_view https://review.openstack.org/487216
21:11:28 jangutter mriedem, jaypipes: should I assert on exception.NovaException or exception.InternalError at https://review.openstack.org/#/c/486426/6/nova/tests/unit/virt/libvirt/test_vif.py@1616
21:14:01 mriedem internal error
21:14:17 mriedem you should assert the thing being raised
21:14:33 jangutter roger, tightening the noose.
21:14:49 mriedem try/catch Throwable
21:15:38 openstackgerrit Jan Gutter proposed openstack/nova master: Add VIFHostDevice support to libvirt driver https://review.openstack.org/486426
21:17:23 jaypipes mriedem: raisable. :P
21:22:52 openstackgerrit Jan Gutter proposed openstack/nova master: Netronome SmartNIC Enablement https://review.openstack.org/483459
21:26:29 mriedem dansmith: fyi about fleet week to the dev list
21:26:31 mriedem sent
21:28:06 dansmith mriedem: ack, I'll read when I return in a bit
21:35:20 mriedem jaypipes: your claims in the scheduler change
21:35:30 mriedem the ironic nv job failed on a scheduling novalidhost
21:35:30 mriedem http://logs.openstack.org/66/483566/10/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/3a475d1/logs/screen-n-sch.txt.gz#_Jul_24_16_26_25_372045
21:35:44 mriedem Jul 24 16:26:25.372045 ubuntu-xenial-osic-cloud1-disk-10046831 nova-scheduler[11679]: DEBUG nova.scheduler.filter_scheduler [None req-76fb384b-64f6-4ed7-86fd-dbb616497e43 tempest-BaremetalBasicOps-2030059131 tempest-BaremetalBasicOps-2030059131] Found host state None that wasn't in allocation requests. Skipping. {{(pid=11679) _schedule /opt/stack/new/nova/nova/scheduler/filter_scheduler.py:196}}
21:38:29 mriedem aha
21:38:31 mriedem YES!
21:40:51 openstackgerrit Ildiko Vancsa proposed openstack/nova master: Implement new attach Cinder flow https://review.openstack.org/330285
21:43:54 ildikov mriedem: I fixed the translate patch and the attach patch too should have a clean test run now
21:44:08 ildikov mriedem: let me know if there's anything else I should do
21:44:13 ildikov mriedem: tnx
21:53:24 openstackgerrit Matt Riedemann proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566
21:53:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Set IronicNodeState.uuid in _update_from_compute_node https://review.openstack.org/487227
21:53:26 mriedem jaypipes: ^
22:04:33 jackie-truong Hi @dansmith, are you available? I have some questions on nova/objects/instance.py that I think you can answer
22:04:46 dansmith jackie-truong: for a few minutes yeah
22:05:22 jackie-truong dansmith: I'm helping out with this patch https://review.openstack.org/#/c/457711/
22:06:10 jackie-truong dansmith: And we're having some issues with properly initializing the list of strings (trusted_certs) so that it is actually an Instance field
22:07:01 dansmith jackie-truong: well, it's defined as a field, which means it's actually a field, but go on...
22:07:38 jackie-truong We're seeing the error "Instance object has no attribute 'trusted_cert'"
22:08:03 dansmith jackie-truong: well, it's defined as "trusted_certs" (note the plural), so maybe that's why?
22:08:23 jackie-truong dansmith: Sorry, that was a typo on my part
22:08:35 jackie-truong "Instance object has no attribute 'trusted_certs' "
22:08:37 dansmith damn, thought that was going to be quick :P

Earlier   Later