| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-23 | |||
| 16:51:24 | dansmith | artom: yes | |
| 16:52:45 | dansmith | I thought the only bits left of the claiming stuff don't actually really interact that much with the resource tracker, | |
| 16:53:01 | dansmith | and rather just see if the things being asked for exist on the compute.. meaning the pci device and the numa arrangement | |
| 16:53:17 | dansmith | so I'm not sure why the reporting stuff is really related directly | |
| 16:54:40 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 16:54:40 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 16:54:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | |
| 16:54:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | |
| 16:54:42 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 16:54:42 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 16:54:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595 | |
| 16:54:47 | efried | dansmith: I should only have to restart n-cpu to pick up a change in oslo.service, yah? | |
| 16:54:53 | efried | at least, the change we care about | |
| 16:55:08 | dansmith | efried: if you're just testing that service then obviously :) | |
| 16:55:35 | efried | I didn't know if other services and their e.g. RPC objects would care. | |
| 16:56:05 | efried | dansmith: Then I've confirmed the oslo.service fix doesn't fix the problem. | |
| 16:56:31 | efried | (which doesn't mean it's wrong, just not complete) | |
| 16:57:17 | dansmith | artom: okay, I see now, I forgot how this works, but I see that if we don't remove that skip we won't do the actual checking of anything, so ignore me | |
| 16:57:20 | dansmith | efried: sweet | |
| 16:58:35 | artom | dansmith, yeah, it's coupling great? | |
| 16:58:40 | artom | *isn't | |
| 16:59:06 | artom | Right, pushed, bottom 4 should be good, now lunch | |
| 17:01:51 | ozzzo | It looks like nova is keeping mysql connections open until they time out. Is there a way to stop these error messages? | |
| 17:02:00 | ozzzo | Aborted connection 10726 to db: 'nova' user: 'nova' host: 'asdf' (Got timeout reading communication packets) | |
| 17:16:08 | mnaser | efried: if you have a few seconds, i wanted to know if you had thoughts about https://review.opendev.org/#/c/676540/ -- should i mark it no longer as deprecated as it's clearly being used? | |
| 17:17:46 | efried | mnaser: or propose whatever alternative we would prefer. mriedem ^ | |
| 17:18:39 | mnaser | i mean if it's clearly being consumed, then i don't mind making the change to stop marking is an deprecated.. | |
| 17:25:43 | openstackgerrit | Merged openstack/nova stable/rocky: Remove experimental job on openSUSE 42.3 https://review.opendev.org/677176 | |
| 17:25:50 | openstackgerrit | Merged openstack/nova stable/queens: Remove experimental job on openSUSE 42.3 https://review.opendev.org/677177 | |
| 17:25:57 | openstackgerrit | Merged openstack/nova stable/pike: Remove experimental job on openSUSE 42.3 https://review.opendev.org/677178 | |
| 17:28:37 | efried | mriedem, dansmith: Posted details on the SIGHUP patch https://review.opendev.org/#/c/641907/ | |
| 17:29:01 | efried | We are indeed breaking the privsep pipe (/me says as if I know what I'm talking about) | |
| 17:29:15 | efried | unclear where that would need to be fixed. | |
| 17:29:42 | mriedem | stephenfin: yeah irrelevant-files | |
| 17:36:47 | sean-k-mooney | stephenfin: add a followup [DNM] patch that add whitespace to a file that would trigger tempest or temporally remove one fo the fiels from irrelevent-files. i generall prefer to have a DNM testing patch as i can just run the job i care about. | |
| 17:37:39 | mriedem | efried: mnaser: i think we likely should undeprecate the direct file download stuff since clearly there are users and those users have made attempts to upstream an alternative solution which hasn't been approved, but not wholesale rejected either from what i understand, so it's probably not good to just rip it out from them when we're not providing an alternative | |
| 17:38:16 | efried | wfm | |
| 17:38:32 | sean-k-mooney | stephenfin: that chante will alter they way devstack install nova as nova installs all packages in requirements.txt and test-requiremnts.txt system wide by default so it should pass in the gate | |
| 17:38:59 | sean-k-mooney | stephenfin: however if you enable installing in virtual envs it may not work anymore | |
| 17:39:53 | sean-k-mooney | stephenfin: without a change to devstack nova would be relying on other pojects to install pymsql or psycopg2 for postgress | |
| 17:43:14 | efried | mriedem: any advice on next steps for copying mysql logs? https://review.opendev.org/#/c/678051/ | |
| 17:45:54 | efried | mriedem: | |
| 17:45:54 | efried | Warning: Permanently added '166.78.116.9' (ECDSA) to the list of known hosts. | |
| 17:45:54 | efried | sudo: no tty present and no askpass program specified | |
| 17:45:54 | efried | rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] | |
| 17:45:54 | efried | rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1] | |
| 17:47:12 | sean-k-mooney | i dont think you want to copy them to the executer liek that i think you want to drop them into the working directory on the vm and have the normal log copy task copy them at the end | |
| 17:48:14 | sean-k-mooney | i may be wrong about that but i think that is what i have seen done before | |
| 17:49:53 | artom | sean-k-mooney, do you know if we would break anything if we started doing claims for all live migrations? I'm thinking of PCI stuff... | |
| 17:50:13 | artom | Presumably claims for PCI aren't broken in and of themselves, since they're used to boot instances and for cold migrations | |
| 17:50:31 | sean-k-mooney | we do claims for pci deivces for migration already | |
| 17:50:40 | artom | *live*? | |
| 17:50:43 | sean-k-mooney | yes | |
| 17:51:07 | sean-k-mooney | well for neutron sriov ports | |
| 17:51:17 | sean-k-mooney | we dont support live migration with other pci devices | |
| 17:51:30 | sean-k-mooney | but for sriov migration we added cliams | |
| 17:51:45 | sean-k-mooney | *sriov live migration | |
| 17:51:51 | artom | Are we talking about the same claims? | |
| 17:51:59 | sean-k-mooney | in the resouce tracker | |
| 17:52:03 | artom | Yeah | |
| 17:52:10 | sean-k-mooney | we do it as part of pre live migration at dest | |
| 17:52:11 | artom | Could you point me to the review/code? | |
| 17:52:22 | sean-k-mooney | lookign for it now | |
| 17:53:30 | sean-k-mooney | https://review.opendev.org/#/c/616120/ | |
| 17:53:41 | sean-k-mooney | and i think we use them in https://review.opendev.org/#/c/620115/28 | |
| 17:54:32 | sean-k-mooney | the pci manager works differently then the numa toplogy in the RT | |
| 17:54:59 | sean-k-mooney | but ehere is the claim in the compute manager | |
| 17:55:01 | sean-k-mooney | https://review.opendev.org/#/c/620115/28/nova/compute/manager.py | |
| 17:55:45 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 17:55:45 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 17:56:43 | artom | sean-k-mooney, it's not actually creating a MoveClaim object though, is it? | |
| 17:57:33 | artom | sean-k-mooney, I'm thinking of a scenario where we live-migration a NUMA instance with neutron SRIOV ports | |
| 17:57:50 | sean-k-mooney | no its not as far as i rememeber | |
| 17:57:53 | artom | Will the MoveClaim for the NUMA part conflict with the PCI claim for the SRIOV part | |
| 17:58:23 | sean-k-mooney | no we claim them in the rt and store the pci addresses in the live migration data object | |
| 17:58:31 | sean-k-mooney | i dont think we are using move claims at all | |
| 17:59:46 | sean-k-mooney | we might use move claims for cold migration i have not check that in a long time | |
| 17:59:52 | mriedem | efried: i think it's failing because the dest doesn't exist when your task runs | |
| 18:00:03 | mriedem | that's done later here https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/5ba0f501-9e58-4f71-859a-ef084a19e0be/#line-11 | |
| 18:00:33 | efried | mriedem: infra said it was because sudo is disabled for tox by default. Latest respin (above) reparents to openstack-tox-with-sudo. | |
| 18:00:33 | artom | sean-k-mooney, if you have the hardware to test those 2 together it would put many minds at east, I think | |
| 18:00:58 | sean-k-mooney | i plan to do that yes | |
| 18:01:20 | sean-k-mooney | ill do it monday or tuesday if you like | |
| 18:01:34 | efried | mriedem: but maybe that too. Though I would have thought they would set up log dirs before invoking posts. | |
| 18:02:50 | sean-k-mooney | i have 1 of my sriov systems setup with 18.04 so i need to install 18.04 on the second one an then install centos on a third host to test something else | |
| 18:03:19 | sean-k-mooney | ill kick of those now and i should be able to test it monday if you bug me or tuesday if you dont | |
| 18:05:39 | artom | sean-k-mooney, appreciated :) | |
| 18:06:30 | mriedem | efried: maybe it is https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/c8a99499-4049-4878-b0a9-13a9292437f4/#line-1 | |
| 18:06:48 | mriedem | efried: so maybe your dest needs to be '{{ zuul_log_path }}' | |
| 18:06:53 | mriedem | er + /mysql/ | |
| 18:07:24 | mriedem | or {{ zuul_output_dir }}/logs/mysql | |
| 18:07:28 | mriedem | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/f31abc72-92ae-4e7d-b37c-8c124f7f32bd/#line-1 | |
| 18:09:55 | efried | mriedem: okay, can try something along those lines if this one borks again. | |
| 18:10:25 | mriedem | ok | |
| 18:10:31 | mriedem | you know about the ara report right? | |
| 18:10:39 | mriedem | to see what runs before and after you with what values | |
| 18:13:52 | sean-k-mooney | artom: once i have it up an running i can give you remote acess to poke around if you want. i have a static ip so i can easily port forwad you to i the hosts. ubuntu is install now so i just need to add a couple of kernel arges to make sriov work then stack with your code | |
| 18:14:07 | artom | sean-k-mooney, ack, that works | |
| 18:15:20 | artom | sean-k-mooney, I'm thinking of this bit in the resource tracker: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L283-L295 | |
| 18:15:35 | artom | For NUMA LM, we call that during check_live_migration_at_destination | |